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Results  from  the  first  part  demonstrated  an  improvement  in  troubleshooting  performance  when 
the  automated  presentation  system  was  used  to  display  troubleshooting  information.  Specifically  it 
was  found  that:  (1)  The  more  difficult  the  problem,  the  more  effective  the  automated  presentation 
system.  (2)  Use  of  the  automated  presentation  system  led  to  significant  reductions  in 
troubleshooting  time  (up  to  k0%)  and  error  rates  (fewer  false  replacements  and  fewer  unnecessary 
tests),  while  resulting  in  a  ICO  percent  success  rate  for  fault  isolation.  (3)  Regardless  of  experience 
level,  technicians  were  more  successful  and  took  less  time  locating  additional  information  when 
using  the  automated  presentation  system.  (9)  Technicians  preferred  an  automated  presentation 
system  over  paper-based  TMs. 

Based  on  the  results  from  the  second  part  of  the  evaluation,  the  authors  conclude  that  the 
transition  from  a  paper  to  electronic  medium  for  presentation  of  technical  information  will  have  a 
major  impact  on  the  Navy's  personnel  subsystem  in  two  important  respects:  (1)  the  way  in  which 
technicians  interact  with  technical  information  contained  in  the  automated  data  base,  and  (2)  the 
changing  role  of  maintenance  technicians  as  they  become  fully  or  even  partially  supported  by  an 
automated  presentation  system.  The  authors  also  conclude  that  for  shipboard  maintenance 
applications,  the  automated  presentation  system  must  be  available  at  dispersed  locations  to  ensure 
adequate  access  and  availability. 


FOREWORD 


This  effort  was  conducted  in  support  of  the  Joint  Service  Manpower  and  Training 
Development  Program  sponsored  by  the  Office  of  the  Under  Secretary  of  Defense, 
Engineering  and  Logistics  Support,  Work  Unit  Code  99000N:  Personal  Electronic  Aid  for 
Maintenance  (PEAM).  The  monitoring  organization  was  the  U.S.  Army  Research  Institute, 
PERI-IP.  The  work  described  in  this  report  involved  the  collection  of  both  objective  and 
subjective  data  for  compari.  g  and  evaluating  a  computer-based  maintenance  information 
delivery  system  as  an  alternative  to  conventional  paper-based  maintenance  manuals. 

Appreciation  is  expressed  to  the  Commanding  Officer,  Combat  Schools  Technical 
Command,  Mare  Island,  California,  and  the  Commander,  Naval  Surface  Forces  Pacific, 
San  Diego,  California,  for  providing  participants.  Appreciation  is  also  expressed  to  Code 
4Y40  of  the  Naval  Ship  Weapon  Systems  Engineering  Station,  which  provided  subject 
matter  experts  during  the  development  of  the  data  base. 

This  report  is  the  fourth  Navy  Personnel  Research  and  Development  Center  report 
addressing  electronic  presentation  of  technical  information:  (1)  NPRDC  Technical  Note 
S7-30  compares  design  features  of  four  developmental  electronic  technical  information 
delivery  systems;  (2)  NPRDC  Technical  Note  87-41  analyzes  troubleshooting  performance 
of  maintenance  technicians  who  used  both  paper-based  maintenance  manuals  and  a 
computerized  information  presentation  system;  and  (3)  NPRDC  Technical  Report  88-6 
compares  alternative  computer-based  print,  audio,  and  graphics  methods  for  presenting 
job  task  instructions.  The  research  in  this  report  is  part  of  a  larger  PEAM  evaluation 
reported  bv  the  Army  Research  Institute  in  Schurman  and  Kincaid  (1988)  and  Wisher  and 
Kincaid  (1988). 

This  report  is  intended  for  use  by  military  and  civilian  personnel  concerned  with 
designing  or  developing  automated  technical  information  presentation  systems.  Com¬ 
ments  and  recommendations  regarding  this  report  are  welcome. 

This  Center's  human  factors  program  and  staff,  including  the  authors  of  this  report, 
were  organizationally  transferred  to  the  Naval  Ocean  Systems  Center,  San  Diego,  in 
March  1988.  Tor  further  information  or.  this  or  related  work,  contact  Dr.  Smillie  at  the 
Naval  Ocean  Systems  Center,  ATTN:  Code  445,  Combat  Systems  Branch,  San  Diego, 
California  92152-5000,  or  call  (619)  553-8015  or  AUTOVON  553-8015. 


B.  E.  BACON  3.  S.  McMICHAEL 

Captain,  U.S,  Navy  Technical  Director 

Commanding  Officer 
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SUMMARY 


Problem 

The  printed  technical  manual  (TM)  is  the  primary  means  for  conveying  the  informa¬ 
tion  needed  to  support  the  operation  and  maintenance  of  virtually  all  systems  and 
equipments  in  the  Navy.  Paper-based  TMs  are  heavy  and  voluminous,  and  their 
development  and  updating  are  both  costly  and  time-consuming.  The  paper  medium  has 
inherent  constraints  inconsistent  with  efficient  organization  of  technical  information.  In 
particular,  information  is  static  and  only  presented  at  a  single  level  of  detail. 

Purpose 

The  primary  purpose  of  this  effort  was  to  conoare  the  efficiency  and  effectiveness 
of  electronically  delivered  technical  information  with  that  of  paper-based  TMs  in 
supporting  the  information  needs  of  Navy  technicians.  A  secondary  purpose  was  to  obtain 
information  related  to  the  implementation  of  an  automated  presentation  system  from 
prospective  users  in  the  surface  fleet. 

Part  I:  Comparative  Assessment 

The  technical  objective  of  Part  1  was  to  evaluate  an  automa'  presentation  system 
as  a  maintenance  aid  for  the  NATO  SEASPARROW  Surface  Mis  ’  system  (NSSMS). 

Method 


Two  troubleshooting  problems  (firing  officer  console  and  director  control)  were 
defined  by  subject  matter  experts  as  representative  of  the  types  of  problems  encountered 
by  NSSMS  fleet  technicians.  A  scenario  was  developed  for  each  problem.  The  two 
delivery  methods  used  to  present  troubleshooting  information  were  a  Sun  model  3/75 
computer,  that  contained  the  data  base,  and  the  standard  technical  manuals  for  the 
NSSMS  MK57  Mods  2  and  3. 

Twenty-eight  NSSMS  "C"  school  in'*. motors  and  students  participated  in  the  study. 
Each  participant  was  randomly  assigned  to  a  two-problem  set.  Combinations  of 
presentation  and  delivery  method  were  balanced  to  avoid  experimental  bias. 

The  independent  variables  of  technician  experience  level,  technical  information 
presentation  method,  and  NSSMS  troubleshooting  problems  were  arranged  in  a  2  X  2  X  2, 
mixed  design.  Data  were  collected  on  1^*  different  time  and  error  measures. 

Results 

A  statistically  significant  interaction  was  obtained  between  the  presentation  method 
and  troubleshooting  problem.  Put  simply,  this  means  that  the  effect  that  the  paper-  and 
computer-based  delivery  systems  had  on  the  performance  measures  was  not  the  same 
across  the  two  troubleshooting  problems  administered.  When  analyses  were  performed  on 
the  performance  measures  for  the  firing  officer  console  problem,  results  showed 
significant  differences  between  the  two  technical  information  delivery  methods.  Use  of 
the  automated  presentation  system  for  the  apparently  more  difficult  firing  officer  console 
problem  resulted  in  a  60  percent  reduction  in  total  troubleshooting  time.  More 
importantly,  there  was  a  100  percent  success  rate  in  fault  isolation  using  the  automated 
presentation  system,  compared  with  a  53  percent  rate  using  the  paper-based  TMs.  None 
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of  the  technicians  using  the  automated  presentation  system  made  false  replacement 
errors,  while  those  using  TMs  averaged  slightly  less  than  one  false  replacement  (0.70)  per 
problem.  Other  error  measure  comparisons  also  supported  the  effectiveness  of  the 
automated  presentation  system.  Technicians  who  used  the  automated  presentation  system 
checked  twice  as  many  proof  points  and  were  12  times  less  likely  to  commit  an  "out-of- 
bounds"  error.  (An  out-of-bounds  error  is  an  incorrect  deduction  that  the  testing  of  one  or 
more  printed  circuit  cards  will  provide  useful  information  for  isolating  the  failure  when, 
in  fact,  the  card  selected  for  testing  is  functioning  properly  and  is  not  contained  within 
the  boundaries  of  the  fault  path.)  Further,  when  the  technicians  used  the  automated 
presentation  system,  they  performed  fewer  tests  (a  ratio  of  1:1.3)  and  had  fewer  invalid 
checks  (a  ratio  of  1:3)  compared  to  when  they  used  the  TMs. 

There  was  only  one  difference  between  the  performance  of  inexperienced  and 
experienced  technicians,  which  occurred  when  the  participants  were  asked  to  locate 
narrative  information  for  the  two  faulty  printed  circuit  cards.  When  using  the  TMs,  the 
experienced  group  had  a  higher  success  rate  (71%)  than  the  inexperienced  group  (33%). 

In  a  user  questionnaire,  the  participants  rated  the  automated  presentation  system  as 
an  improvement  over  TMs  for  the  NSSMS  in  terms  of  efficiency  and  effectiveness  in 
providing  needed  maintenance  information. 

Part  H:  Implementation  Issues 

The  technical  objective  of  Part  II  was  to  obtain  information  related  to  fleet 
implementation  of  an  automated  presentation  system. 

Method 

Information  was  collected  from  34  people--25  NSSMS  technicians,  7  NSSMS  "C" 
school  instructors,  an  NSSMS  documentation  expert,  and  the  project  director  for  the 
development  of  an  automated  data  base  for  the  MK  15  (PHALANX)  Close-In  Weapon 
System.  Questionnaires  and  open-ended  interviews  were  used  to  collect  the  data  on 
(1)  implementation  concerns  of  the  various  anticipated  users,  and  (2)  impact  of  innovative 
changes  on  the  Navy's  personnel  subsystem. 

Results 

A  consensus  was  reached  on  several  Issues  related  to  the  implementation  of  an 
automated  technical  information  delivery  system.  For  example,  it  was  felt  that  multiple 
delivery  devices  and  copies  of  the  data  base  should  be  located  throughout  the  weapon 
system  they  support  to  ensure  adequate  coverage,  availability,  and  survivability.  Backup 
copies  of  the  media  used  to  store  the  automated  technical  information  data  base  were 
also  considered  to  be  a  necessity  for  shipboard  applications. 

Revisions  and  updates  should  be  fast  and  easy  to  perform,  resulting  in  completely 
revised  "manual  sets"  of  the  automated  system's  data  base  using  magnetic  or  optical 
media. 

While  a  need  for  training  in  the  operation  of  the  automated  presentation  system  itself 
was  noted,  the  automated  presentation  system  could,  in  turn,  be  used  to  support  both  on- 
the-job  and  participatory  shipboard  training. 


Since  many  advanced  weapon  systems  in  the  Navy  contain  built-in  test  (BIT) 
capabilities,  the  power  and  advantages  of  an  automated  presentation  system  would  not  be 
fully  realized  if  it  were  restricted  to  a  single  interface;  that  is,  between  the  user  and  the 
technical  information  data  base.  Accordingly,  it  was  felt  that  an  interface  should  be 
developed  that  would  allow  the  automated  presentation  system  to  initiate  BIT  routines 
and  process  BIT  data  directly.  With  this  "expert  reasoning"  capability,  the  automated 
system  could  then  evaluate  BIT  results  and  provide  the  technician  with  a  menu  of 
alternative,  most  likely,  next  actions  to  perform;  thereby  minimizing  the  role  of  the 
technician  as  a  data  transfer  agent  and  maximizing  that  person  as  an  analytic  decision 
maker.  Interfaces  with  other  shipboard  data  bases  have  to  be  considered,  such  as  the 
shipboard  non-tactical  ADP  program  (SNAP),  the  consolidated  automatic  support  system 
(CASS),  etc.,  to  provide  maximum  usage  of  available  maintenance  data  and  to  simplify 
parts  ordering  and  documentation  of  maintenance  actions. 

Conclusions 

To  summarize,  troubleshooting  performance  was  improved  when  the  automated 
presentation  system  was  used  to  display  troubleshooting  information  for  the  NATO 
SEASPARROW  Surface  Missile  System.  Specifically: 

).  The  more  difficult  the  problem,  the  more  effective  the  automated  presentation 
system. 

2.  Use  of  the  automated  presentation  system  led  to  significant  reductions  in 
troubleshooting  time  (up  to  40%)  and  error  rates  (fewer  false  replacements  and  fewer 
unnecessary  tests),  while  at  the  same  time  resulting  in  a  100  percent  success  rate  for 
fault  isolation. 

3.  Regardless  of  experience  level,  technicians  were  more  successful  and  took  less 
time  locating  additional  information  when  using  the  automated  presentation  system. 

4.  Technicians  preferred  an  automated  presentation  system  over  existing  paper- 
based  TMs. 

The  authors  conclude  that  the  transition  from  a  paper  to  electronic  medium  for 
presentation  of  technical  information  will  have  a  major  impact  on  the  Navy's  personnel 
subsystem  in  two  important  regards:  (1)  the  way  in  which  technicians  interact  with 
technical  information  contained  in  the  automated  data  base,  and  (2)  the  changing  role  of 
maintenance  technicians  as  they  become  fully  or  even  partially  supported  by  an 
automated  presentation  system.  The  authors  also  conclude  that  for  shipboard 
maintenance  applications,  the  automated  presentation  system  must  be  available  at 
dispersed  locations  to  ensure  adequate  access  and  survivability. 
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INTRODUCTION 


Problem 

The  printed  technical  manual  (TM)  is  the  primary  means  for  conveying  the  informa¬ 
tion  needed  to  support  the  operation  and  maintenance  of  virtually  all  systems  and 
equipments  in  the  Navy.  Paper-based  TMs,  however,  are  heavy  and  voluminous. 
Constraints  inherent  within  the  paper  medium  are  inconsistent  with  efficient  organization 
of  the  information.  In  addition,  the  technical  information  is  static  and  only  presented  at  a 
single  level  of  detail.  Finaily,  1  id  development  and  updating  are  both  costly  and  time- 
consuming. 

Volume  and  weight  of  paper  TMs  have  reached  unwieldy  proportions.  Duffy  (1985) 
noted  that  a  military  aircraft  introduced  in  1950  required  1,800  pages  to  document  the 
entire  operation  and  maintenance  system.  By  1975,  that  figure  had  increased  14,000 
percent  to  260,000  pages  for  the  Navy's  F-14  fighter.  The  technical  documentation  to 
support  the  F/A-1S  fighter  is  in  excess  of  700,000  pages.  In  a  recent  field  survey, 
representatives  from  the  Space  and  Naval  Warfare  Systems  Command  determined  the 
weight  and  volume  of  paper-based  information  stored  aboard  different  types  of  Navy 
platforms  (Space  and  Naval  Warfare  Systems  Command,  1987).  Results  showed  that  the 
weight  and  volume  of  paper  and  associated  storage  containers  aboard  the  USS 
VINCENNES  (CG-49)  totaled  71,503  pounds  and  1,806  cubic  feet.  This  weight  is 
equivalent  to  142  pounds  for  each  man  assigned  to  the  Aegis-class  cruiser. 

Searching  through  multiple  volumes  of  TMs  to  obtain  needed  information  can  become 
a  frustrating  task.  A  dramatic  example  of  this  access  problem  was  described  in  a  report 
issued  by  the  Ceneral  Accounting  Office  (1979).  The  report  noted  that  to  isolate  and 
repair  one  particular  radar  malfunction  on  a  C-141  aircraft,  technicians  had  to  refer  to, 
and  cross-reference,  165  pages  in  8  separate  documents.  An  unfortunate  by-product  of 
this  access  problem  is  that  it  tends  to  discourage  users  from  following  maintenance 
procedures  specified  in  the  TM.  Rue  and  Lorenz  (1983)  reported  that  the  bypassing  of 
critical  procedural  steps  in  the  TM  was  a  major  factor  in  the  unnecessary  removal  of  line 
replaceable  units  on  Air  Force  avionics  equipment.  Specifically,  their  study  found  that  13 
percent  of  the  1,008  unnecessary  replacements  of  non-faulty  components  was  attributed 
to  ineffective  TMs  (e.g.,  troubleshooting  procedures  were  too  difficult  to  follow,  or  took 
too  long  to  perform,  or  were  inaccurate). 

Another  major  shortcoming  of  paper -based  TMs  is  that  the  information  is  static. 
Hence,  the  TM  is  not  readily  adaptable  to  the  needs  of  the  user.  For  example,  an 
inexperienced  technician  (or  one  who  has  not  worked  on  a  particular  price  of  equipment 
for  some  time)  may  require  more  detailed  information  than  is  provided  in  the  TM  to 
perform  such  funr  .ions  as  locating  test  points  and  removing,  replacing,  or  adjusting 
components.  Conversely,  a  highly  experienced  technician  may  have  performed  a 
particular  maintenance  task  so  often  that  little  more  is  needed  than  a  checklist  to  ensure 
that  no  maintenance  step  is  overlooked.  In  addition,  the  inherent  structure  of  paper-based 
TMs  prevents  a  dynamic  interactive  exchange  between  the  information  source  and  user. 
For  example,  TMs  lack  the  capability  to  (1)  provide  hands-free  access  to,  or  altei  native 
methods  for  delivery  of,  needed  technical  information  (e.g.,  voice  input  or  text-to-speech 
output);  (2)  highlight  or  animate  selected  portions  of  the  technical  information  on  demand 
(e.g.,  emphasizing  the  physical  location  of  components  using  flashing  arrows  or 
highlighting  the  path(s)  of  a  signal  across  multiple  pages  of  schematic  diagrams);  or  (3) 
provide  direct,  on-line  access  to  computerized  data  banks  (e.g.,  spare  parts  supply 
information  or  historical  records  of  equipment  casualties  and  their  solutions). 
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Finally,  paper-based  TMs  have  large  time  and  dollar  costs  associated  with  production, 
distribution,  updating,  and  correction.  In  1978,  the  Air  Force  estimated  th3t  it  spent  $70 
million  per  year  to  add  new  manuaL  or  revise  existing  ones  (Ceneral  Accounting  Office, 
1979).  More  recently,  it  has  been  estimated  that  the  per  page  development  cost  for  TMs 
within  the  Department  of  Defense  ranged  from  $250  for  a  standard  manual  to  more  than 
four  times  tha:  for  user-oriented  ones  (Duffy,  1985).  Beyond  cost  considerations  are  the 
inherent  delays  in  updating  TMs  to  reflect  modifications  in  hardware  installation  and 
engineering  changes,  or  to  correct  errors  and  discrepancies  identified  by  the  TM  users.  In 
regard  to  the  latter,  Chenzoff  and  Joyce  (1983)  reported  that  errors  in  TMs  that  affect 
the  safety  of  the  user  are  processed  quickly,  in  as  short  a  period  as  2  wee’<s.  Lower 
priority  changes,  on  the  other  hand,  require  TM  originators  to  submit  advrnce  change 
notices  within  90  days  and  subsequent  delivery  of  actual  changes  within  six  months. 

In  an  attempt  to  remedy  the  foregoing  deficiencies  within  the  paper  medium,  the 
Office  of  the  Secretary  of  Defense  (OSD)  recommended  that  the  services  initiate  research 
and  development  efforts  aimed  at  improving  the  overall  quality,  management,  and 
delivery  of  military  technical  information.  One  outgrowth  of  this  OSD  recommendation 
was  the  development  of  a  portable  computer-based  technical  information  delivery  system 
called  the  Personal  Electronic  Aid  for  Maintenance  (PEAM). 

Background 

The  PEAM  project  began  as  an  exploratory  development  effort  jointly  sponsored  by 
the  Naval  Training  Systems  Center  (NTSC)  and  the  Army  Project  Manager  for  Training 
Devices.  The  initial  effort,  conducted  from  FYSO  to  FY82,  resulted  in  the  formulation  of 
a  conceptual  definition  for  PEAM  together  with  detailed  specifications  for  developing  its 
computer  hardware  and  software  systems.  Also,  during  FY82,  a  memorandum  of 
understanding  was  signed  by  the  Commander,  Army  Research  Institute  and  the  Command¬ 
ing  Officer,  NTSC.  The  memorandum  established  agreements,  support,  and  resource 
responsibilities  for  the  full-scale  engineering  development  of  PEAM.  The  latter  effort 
culminated  with  the  delivery  of  four  PEAM  prototype  devices  from  the  contractor  in 
FY86.  (See  Schurman  arid  Kincaid  (1988)  for  a  detailed  description  of  the  physical  and 
operational  features  of  the  PEAM  prototype.) 

Concurrent  with  the  PEAM  engineering  development,  OSD  tasked  the  services  with 
the  responsibility  to  test  and  evaluate  the  PEAM  concept  as  an  alternative  to  paper  TMs. 
In  FY85,  OSD  tasked  both  the  Army  Research  Institute  and  the  Navy  Personnel  Research 
and  Development  Center  to  design  and  conduct  field  tests  of  PEAM  in  their  respective 
services.  This  report  describes  the  Navy  test  and  evaluation.  (See  Schurman  and  Kincaid 
(1988)  for  the  Army  test  and  evaluation.) 

Purpose 

The  primary  purpose  of  this  effort  was  to  compare  the  efficiency  and  effectiveness 
of  electronically  delivered  technical  information  with  that  of  paper-based  TMs  in 
supporting  the  information  needs  of  Navy  technicians.  A  secondary  purpose  was  to  obtain 
information  related  to  the  implementation  of  the  PEAM  delivery  concept  from  prospec¬ 
tive  users  in  the  surface  fleet. 

Accordingly,  a  two-part  approach  was  followed.  The  first,  Comparative  Assessment, 
involved  both  quantitative  and  qualitative  assessments  of  the  PEAM  concept  under 


controlled  test  conditions.  The  second  Implementation  Issues.  addressed  issues  and 
problem  areas  to  be  considered  prior  to  the  formal  introduction  of  s  PEA V -like  device  in 
the  fleet. 


PART  Is  COMPARATIVE  ASSESSMENT 

The  technical  objective  of  Part  I  was  to  test  PEAM  in  an  operational  environment  as 
a  maintenance  aid  for  the  NATO  SEASPARROW  Surface  Missile  System  (N'SSMS).  As 
initially  planned,  comparisons  were  to  be  made  between  PEAM  and  paper-based  documer 
tation  in  supporting  both  preventive  and  corrective  maintenance  activities  on  the  NSSMu 
(see  Appendix  A).  Several  technical  difficulties  were  encountered,  however,  in  loading 
the  N'SSMS  data  base  onto  PEAM  memory  modules,  which  effectively  precluded  (1)  use  of 
the  PEAM  hardv/are  at  d  (2)  data  collection  in  a  shipboard  environment.  Table  1  lists  a 
chronology  of  the  PEAM  hardware  and  software  difficulties  encountered  when  attempting 
to  load  t!  e  NSSM5  data  base.  As  a  result,  two  major  modifications  were  made  to  the 
original  test  plan  for  the  Navy  evaluation  of  PEAM.  The  first  was  the  use  of  alternative 
computer  hardware  for  presenting  the  technical  information  contained  in  the  NSSMS  data 
base.  The  second  was  the  elimination  of  the  shipboard  comparative  assessment  between 
PEAM  and  existing  documentation  that  supported  scheduled  maintenance  activities.  The 
latter  modification  was  required  because  the  weight  and  bulk  of  the  alternative  computer 
did  not  allow  it  to  be  transported  to  NSSMS  equipment  sites  aboard  ships. 

It  should  also  be  noted  that,  unlike  the  PEAM  device,  the  computer  hardware  used  to 
present  the  NSSMS  data  base  did  not  have  speech  input  or  output  capabilities.  The  utility 
of  these  capabilities,  therefore,  could  not  be  assessed  in  the  present  effort.  (See 
Schurman  and  Kincaid  (19S8)  for  a  description  of  PEAM  hardware  use  and  data  collection 
in  an  operational  environment.) 

Troubleshooting  Problems  and  Administration 

There  were  two  troubleshooting  problems  defined  by  subject  matter  experts  as 
representative  of  the  types  of  problems  encountered  by  NSSMS  fleet  technicians. 
Scenarios  were  then  developed  for  each  problem  using  a  simulation  technique  developed 
by  Connor  (1987).  Tne  problems,  which  were  simulated  on  a  personal  computer  (PC), 
consisted  cf  two  NSSMS  printed  circuit  card  malfunctions— one  affecting  the  firing  officer 
console;  the  other,  the  director  control  unit. 

At  the  start  of  each  problem,  the  major  casualty  sympt  ms  and  operational 
discrepancies  associated  with  the  printed  circuit  card  failure  were  displayed  on  the  PC, 
This  information  was  followed  by  the  presentation  oi  various  menus  from  which 
technicians  were  expected  to:  (1)  select  a  init  >5  the  NSSMS  to  initiate  troubleshooting 
actions,  (2)  specify  a  reference  designation  for  individual  test  points,  card  pins,  com¬ 
ponent  parts,  etc.,  to  be  checked,  (3)  choose  a  particular  type  of  measurement  (e.g., 
voltage,  waveform,  continuity)  to  be  performed  on  a  reference  designation  selected  for 
testing,  (4)  troubleshoot  the  problem  until  such  time  as  the  failure  was  isolated  to  a 
suspected  printed  circuit  card  and  component  part,  and  (5)  replace  the  suspected  faulty 
printed  circuit  card  to  determine  if  that  action  ccrt  acted  the  malfunction.  Figure  I 
provides  an  example  of  the  general  structure  of  the  various  menus  and  data  inputs  used 
for  troubleshooting. 
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Table  1 


Summary  of  Problems  Encountered,  Corrective  Actions  Taken, 
and  Time  Delays  Experienced  when  Attempting 
to  Field  an  Operable  PEAM  Device 


Time  Delay 

Corrective  Action  (duration) 

1.  All  attempts  to  compile  Navy  Developed  software  tools  and  90  Days 

PEAM  data  base  were  unsuccess-  techniques  to  correct  errors;  re- 

ful.  More  than  1000  software  sultant  data  base  now  well  with- 

errors  (e.g.,  misspellings,  im-  in  limits  for  presentation  on 

proper  linkages,  incorrect  syn-  PEAM  hardware, 

tax)  were  identified;  correction 
of  these  errors  alone  exceeded 
memory  capacity  for  presenta¬ 
tion  on  PEAM  hardware. 

2.  Revised  data  base  was  compiled 
successfully  on  SUN  3/75  com¬ 
putet  workstation;  however,  at¬ 
tempts  to  download  to  PEAM 
hardware  failed. 

3.  PEAM  developer  attempted  to 
identify  problems  within  data 
base  and  successfully  compiled 
and  downloaded  a  subset  of  the 
data  base.  Government  down¬ 
loading  attempts  continued  to 
fail.  Apparent  problem  involved 
computer  hardware  differences 
between  government  and  PEAM 
developer. 

Note.  Approximately  7.5  months  were  spent  in  attempting  to  resolve  difficulties  with 

PEAM  hardware  and  software. 


Inability  to  download  data  base  60  Days 

probably  resulted  from  errors  in 
transfer  utility.  PEAM 

developer  provided  new  copy  of 
transfer  utility. 

Used  Sun  model  3/75  computer  60  Days 

workstation  to  present  Navy 

data  base  originally  developed 

for  PEAM  hardware;  revised  test 

plan. 
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Figure  1.  An  example  of  the  menus  used  to  guide  NSSMS  troubleshooting  tasks. 


The  simulation  software  also  presented  feedback  information  to  the  technician 
consisting  of  measurements  obtained  or  waveforms  present  at  particular  test  points, 
prompts  for  data  inputs,  and  messages  concerning  the  validity  of  various  troubleshooting 
and  repair  actions  performed.  In  addition,  the  computer  was  programmed  to  record  data 
from  the  technician's  troubleshooting  performance. 

Technical  Information  Delivery  Systems 

The  two  delivery  methods  used  to  present  troubleshooting  information  were  a  Sun 
model  3/75  computer  that  contained  the  data  base  originally  developed  for  PEAM,  and 
standard  TMs  for  the  NSSMS  MK57  Mods  2  and  3.  The  latter  publications  were  all 
unclassified  and  consisted  of  more  than  40  separate  binders  and  16,000  pages.  With  the 
exception  of  minor  differences  between  the  Sun  computer  and  PEAM  hardware  in  the 
method  of  accessing  information  (i.e.,  use  of  a  mouse  control  versus  pushbutton  or  voice 
activation  of  operator  commands),  the  organization  and  format  of  the  NSSMS  data  base 
were  identical  to  that  which  would  have  been  presented  on  the  PEAM  units. 

To  construct  the  automated  data  base  for  the  NSSMn,  information  from  the  TM  was 
first  organized  into  a  hierarchy  of  expected  maintenance  actions.  This  hierarchy  was 
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modeled  after  the  troubleshooting  strategy  taught  in  the  Navy's  electrical  and  electronics 
training  schools.  That  is,  once  a  malfunction  has  been  detected  at  the  system  level,  the 
data  base  allows  the  technician  to  trace  the  symptoms  of  the  malfunction  through  the 
subsystem,  equipment,  and  component  levels.  In  addition,  branches  to  auxiliary  types  of 
technical  information  are  provided  at  each  level  within  the  hierarchy. 

Next,  the  order  in  which  the  technical  information  had  to  be  stored  in  the  computer 
was  established  and  initial  links  among  various  elements  in  the  data  base  were  deter¬ 
mined.  Individual  troubleshooting  frames  were  then  constructed  from  existing  signal  flow 
diagrams  in  which  signal  inputs  and  outputs  were  identified  and  linked  either  to  previous 
or  subsequent  troubleshooting  frames  in  the  data  base.  Each  troubleshooting  frame  also 
provided  automatic  links  to  supporting  technical  data  (e.g.,  simplified  signal  flow 
diagrams,  parts  information,  narrative  descriptions  of  circuit  functions  and  their  inter¬ 
relationships).  Figure  2  illustrates  how  the  signal  flow  diagrams  were  broken  up  into 
frames  and  how  the  input/output  signals  were  linked  to  other  frames.  Figure  2  also  shows 
how  the  different  types  of  data  were  linked  in  the  automated  data  base.  Appendix  B 
presents  examples  of  the  structure  and  format  of  actual  frames  that  comprised  the 
automated  data  base  for  the  NSSMS. 

This  type  of  approach  to  the  data  base  structure  provides  the  flexibility  necessary  to 
go  beyond  electronic  page  turning.  With  linkages  built  into  the  data  base,  the  user  only 
has  to  decide  what  type  of  information  is  necessary.  Branching  is  automatic.  With  paper, 
the  user  has  to  physically  locate  the  desired  information,  a  very  time-consuming  and 
frustrating  process  when  the  data  base  comprises  several  thousand  pages  of  information. 
For  example,  when  using  TMs  for  one  of  the  NSSMS  troubleshooting  scenarios,  the 
technician  had  to: 

•  start  with  a  symptom  table  in  one  TM  that  referenced 

•  a  second  table  (2^2  pages  later)  that  referenced 

•  a  third  table  (38  pages  later)  that  referenced 

•  a  different  TM  for  a  specific  signal  flow  diagram  that  spanned  10  double-sized 
pages. 

Collectively,  there  were  26  cross-references  to  needed  technical  information  in  the 
foregoing  tables  and  diagrams. 

Test  Participants 

There  were  2S  participants  in  the  study,  consisting  of  15  instructors  and  7  students 
from  the  NSSMS  class  "C"  maintenance  training  course  at  the  Combat  Systems  Technical 
Schools  Command  in  Mare  Island,  California,  and  6  NSSMS  technicians  assigned  to  fleet 
units  in  San  Diego,  California.  These  participants  were  classified  into  two  groups 
consisting  of  20  experienced  and  8  inexperienced  technicians.  These  classifications  were 
based  on  a  combinaticn  of  relevant  field  experience  in  the  maintenance  of  the  NSSMS 
(i.e.,  1  or  more  years  for  the  experienced  group;  less  than  1  year  for  the  inexperienced 
group)  and  judgments  of  immediate  work  supervisors  concerning  the  technicians'  main¬ 
tenance  work  qualifications. 

The  decision  to  include  maintenance  experience  as  a  factor  in  the  present  study  was 
made  for  two  reasons.  First,  it  was  the  contention  of  the  project  team  that  any 
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Figure  2.  An  example  of  a  signal  flow  diagram  from  the  NS5M5 
technical  manual  (a),  and  how  it  was  represented  by 
separate  frames  in  the  automated  presentation  system 
(b).  The  lines  and  arrows  in  the  bottom  figure  depict 
linkages  of  signal  lines  between  frames  or  the  types  of 
support  data  available  from  any  one  frame. 
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computer-based  technical  information  delivery  system  developed  for  military  applications 
must  be  both  useful  to  and  usable  by  technicians  of  varying  knowledge  and  skill  levels. 
Second,  several  studies  have  shown  that  when  evaluating  job  task  proficiency,  level  of 
experience  accounts  for  substantial  variance  in  task  performance  (e.g.,  Brown,  1964;  Goff, 
Schlesinger,  &  Parlog,  1969;  Kieras,  Tibbits,  Sc  Bovair,  1984;  Nugent,  1988). 

Participation  in  the  testing  was  voluntary.  No  one  declined  to  participate,  and  all 
technicians  appeared  to  be  interested  and  cooperative  throughout  the  test  sessions. 

Procedure 


A  detailed  test  proctor’s  guide  was  developed  to  standardize  the  presentation  of 
instructions,  technical  information  delivery  systems,  and  troubleshooting  problems  to  all 
participants.  The  following  sections  summarize  the  procedures  used  in  each  test  session. 


Preliminary  Instructions  and  Procedures 


At  the  start  of  the  session  participants  were  assigned  to  groups  of  two,  irrespective 
of  their  experience  level.  Instructions  were  then  read  aloud  to  the  participants  to  provide 
them  with  a  general  orientation  to  the  testing  and  data  collection  methods.  Procedures 
for  maintaining  the  technician's  anonymity  and  voluntary  participation  in  the  testing  were 
described,  followed  by  a  set  of  questions  regarding  each  participant’s  background. 


The  participants  then  received  instruction  and  practice  on  the  basic  operation  of  the 
computer  hardware  used  to  present  the  simulated  troubleshooting  problems  and,  similarly, 
for  accessing  technical  information  from  the  NSSMS  data  base.  This  training  was 
followed  by  a  general  question-and-answer  session  during  which  time  the  participants 
were  encouraged  to  request  information  and  details  for  any  procedures  that  may  have 
been  unclear  to  them.  Upon  completion  of  the  preliminary  instructions  and  procedures, 
which  averaged  40  minutes,  technicians  were  given  a  short  rest  break. 

Problem  Administration 

The  troubleshooting  problems  were  administered  individually  to  the  participants  at 
one  of  two  workstations.  One  of  these  workstations  contained  standard  maintenance 
publications  for  the  NSSMS;  the  other,  the  automated  NSSMS  delivery  system,  that  is,  the 
Sun  model  3/75  computer.  Both  workstations  had  a  PC  that  hosted  the  simulation 
scenario  for  the  NSSMS  troubleshooting  problems.  Since  the  technical  information  needed 
to  solve  the  troubleshooting  problems  was  available  at  both  workstations,  each  participant 
was  randomly  assigned  one  of  the  two  problems.  Combinations  of  problem  presentation 
and  delivery  method  were  balanced  to  avoid  experimental  bias. 


Before  starting  each  problem,  the  test  participants  were  informed  that  the  test 
administrator  at  each  workstation  would  only  provide  information  to  them  regarding: 
(1)  expiration  of  the  time  allowed  for  completing  each  problem,  or  (2)  actions  needed  to 
extricate  themselves  from  difficulties  encountered  when  using  either  the  PC  or  the  Sun 
computer  (i.e.,  for  inputting  data  or  accessing  technical  information,  respectively).  Upon 
completion  of  the  first  problem,  participants  were  given  a  short  rest  break,  after  which 
they  were  directed  to  the  other  workstation  for  administration  of  the  second  trouble¬ 
shooting  problem. 
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Data  Collection  Forms 


Two  basic,  forms  were  used  to  collect  performance  and  attitudina!  data  from  the  test 
participants. 

Performance  Observation  Form.  This  form,  which  consisted  of  four  sections,  was 
completed  by  the  test  administrator  at  each  workstation.  Section  I  provided  spaces  for 
recording  information  regarding  the  technician’s  ability  to  locate  and  use  correct 
reference  material  for  initiating  troubleshooting  actions  (e.g.,  fault  directories  and 
troubleshooting  indices  in  NSSMS  publications;  equipment  lists  and  fault  menus  in  the 
automated  NSSMS  data  base).  Section  II  was  used  to  record  both  general  and  highly 
specific  information  related  to  the  test  problems  (e.g.,  the  number,  type,  and  validity  of 
various  tests  and  actions  performed).  Essentially,  this  section  was  designed  to  provide  a 
"backup"  record  of  the  participant's  performance  data  in  the  event  that  the  automated 
performance  monitoring  software  developed  for  the  PC  malfunctioned.  Sections  III  and  IV 
were  used  to  record  the  technician's  ability  to  locate  parts  data  and  descriptions  of  the 
physical  and/or  functional  characteristics  of  the  faulty  printed  circuit  cards,  respectively. 
As  a  minimum,  all  four  sections  of  the  performance  observation  form  provided  spaces  for 
recording  start  and  stop  times,  whether  correct  technical  information  was  located  and 
used,  and  observations  or  notes  of  the  test  administrator. 

User  Questionnaire.  This  form  (shown  in  Appendix  C)  was  administered  to  all  test 
participants  immediately  after  the  second  troubleshooting  problem  had  been  completed. 
It  contained  questions  requiring  participants  to:  (1)  rate  the  physical  features  of  the 
computer  used  to  present  the  NSSMS  data  base,  (2)  rate  the  operation  and  software 
features  of  the  automated  presentation  format,  and  (3)  compare  the  automated 
presentation  system  with  the  paper  TMs.  A  fourth  section  took  the  form  of  an  open-ended 
question  for  recording  any  comments,  complaints,  or  suggestions  regarding  the  automated 
presentation  system. 

Debriefing 

The  test  session,  which  lasted  an  average  of  3  hours  per  technician,  ended  with  a 
discussion  of  the  purpose  of  the  study  together  with  a  review  of  the  participant's 
performance  on  the  problems.  The  technicians  were  then  thanked  for  participating  and 
asked  to  refrain  from  discussing  details  of  the  study  with  others  who  had  not  yet  been 
tested. 

Experimental  Hypotheses  and  Design 

The  following  experimental  hypotheses  served  as  the  framework  for  evaluating 
troubleshooting  performance  as  a  function  of  the  technicians'  experience  level  and  the 
two  methods  of  presenting  technical  information: 

1.  Total  task  time  for  troubleshooting  will  be  less  when  the  automated  presentation 
system  is  used. 

Troubleshooting  using  paper-based  manuals  typically  requires  the  acquisition  of 
information  from  various  locations  within  a  single  manual  or  in  two  or  more  manuals. 
Simply  "thumbing"  through  these  manuals  is  highly  time-consuming.  The  automated 
presentation  permits  virtually  immediate  access  to  any  desired  information  and  will 
reduce  total  task  time. 
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2.  Fewer  errors  will  be  made  when  the  automated  presentation  system  is  used. 

Use  of  technical  information  does  not  guarantee  successful  fault  isolation.  A 
critical  factor  in  troubleshooting  when  using  paper-based  TMs  is  the  necessity  to  refer  to 
a  specific  line  or  item  entry  in  a  long  list  or  complex  diagram.  The  likelihood  of  the 
technician  making  a  locational  error  leading  to  unsuccessful  fault  isolation  or  a  false 
replacement  is  relatively  high.  (A  false  replacement  is  the  identification  and  replacement 
of  a  component  that  was  suspected  as  the  cause  of  the  problem  but  did  not  alter  the 
original  set  of  symptoms.)  The  automated  presentation  system,  by  displaying  better 
organized  technical  information  with  easy  access  to  additional  support  information,  will 
increase  successful  fault  isolation  and  decrease  false  replacements. 

3.  When  using  the  automated  presentation  system,  technicians  will  refer  to 
technical  information  more  and,  subsequently,  make  fewer,  more  valid  tests. 

Technicians  tend  to  employ  shortcuts,  presumably  because  of  difficulty  in 
accessing,  understanding,  and/or  using  technical  information  in  its  paper-based  format. 
Since  the  automated  presentation  system  provides  ready  access  to  all  information  in  a 
more  understandable  and  usable  form,  technicians  will  avoid  shortcut  approaches  and 
make  more  use  of  available  information  when  using  the  automated  presentation  system. 
Thus,  there  will  be  fewer  tests  and  the  selection  and  execution  of  tests  will  be  more 
appropriate  (i.e.,  valid)  for  systematically  and  conclusively  isolating  the  fault. 

b.  Inexperienced  technicians  using  the  automated  presentation  system  will  trouble¬ 
shoot  as  well  as  experienced  technicians  using  TMs. 

In  other  words,  the  advantages  of  the  automated  presentation  system  will 
counterbalance  experience  advantages  when  comparing  inexperienced  technicians  using 
the  automated  system  with  experienced  technicians  using  paper  TMs. 

5.  hen  comparing  performance  within  each  technical  information  delivery  system, 
experienced  technicians  will  troubleshoot  better  than  inexperienced  technicians. 

On  a  composite  grading  system  that  includes  time,  errors,  etc.,  it  is  expected 
that  experienced  technicians  will  outperform  inexperienced  technicians  within  each 
information  delivery  sv'tem,  due  to  their  more  thorough  exposure  to  and  working 
knowledge  of  N5SM5  maintenance. 

The  independent  variables  of  technician  experience  level,  technical  information 
presentation  method,  and  NSSMS  troubleshooting  problems  were  arranged  in  a  2  X  2  X  2, 
mixed  design.  While  a  technician's  experience  level  was  fixed,  all  participants  used  both 
presentation  methods.  Combinations  of  troubleshooting  problem  and  presentation  method 
were  balanced  to  avoid  experimental  bias.  The  dependent  measures  were: 

1.  Number  of  out-of-bound  errors--an  incorrect  deduction  that  the  testing  of  one  or 
inore  printed  circuit  cards  would  provide  useful  information  for  isolating  the  failure  when, 
in  fact,  the  card  selected  for  testing  was  functioning  properly  and  was  not  contained 
within  the  boundaries  of  the  fault  path. 

2.  Number  of  valid  checks  performed--the  selection  of  an  appropriate  electrical  or 
electronic  measurement  technique  for  a  particular  test  point,  card  pin,  component  part, 
etc. 
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3.  Number  of  invalid  checks  performed— the  selection  of  an  inappropriate  electrical 
or  electronic  measurement  technique;  for  example,  attempting  to  measure  the  frequency 
of  a  resistor. 

4.  Number  of  proof  points  checked  (i.e.,  the  number  of  possible  input  and  output 
points  tested  in  the  faulty  circuit  that  conclusively  isolates  the  fault). 

3.  Number  of  false  replacements  recommended  to  correct  the  failure— a  tech¬ 
nician's  recommendation  for  replacing  one  or  more  printed  circuit  cards  that  were 
functioning  properly  and  would  not  correct  the  fault  condition. 

6.  Total  number  of  fault  isolation  tests  performed. 

7.  Overall  time  to  problem  completion. 

8.  Success  or  failure  in  isolating  the  faulty  printed  circuit  cards. 

9.  Time  to  initiate  specific  troubleshooting  actions,  that  is,  the  elapsed  time  from 
the  receipt  of  the  equipment  failure  symptoms  to  the  participant's  first  use  of  either  the 
TMs  or  the  automated  presentation  system. 

10.  Success  in  locating  appropriate  diagnostic/troubleshooting  information  in  either 
the  TM  or  the  automated  presentation  system. 

1  1.  Time  to  locate  parts  information  for  faulty  printed  circuit  cards. 

12.  Success  in  locating  the  parts  information  for  faulty  printed  circuit  cards. 

13.  Time  to  locate  narrative  information  describing  the  physical  and/or  functional 
characteristics  of  faulty  printed  circuit  cards. 

14.  Success  in  locating  narrative  information  describing  the  physical  and/or  func¬ 
tional  characteristics  of  faulty  printed  circuit  cards. 

Dependent  measures  1-S  were  categorized  as  those  related  to  troubleshooting,  while 
measures  9-14  were  categorized  as  those  related  to  information  location. 

Results 

PemoRraphic  Comparisons 

Prior  to  conducting  tests  of  the  experimental  hypotheses,  the  authors  conducted 
analyses  to  determine  the  extent  to  which  the  two  groups  of  test  participants  differed 
with  respect  to  items  contained  on  the  personal  background  form.  These  analyses  were 
performed  to  provide  quantitative  evidence  that  the  two  groups  did  differ  on  variables 
related  to  the  experience  level.  Accordingly,  a  multiple  analysis  of  variance  (MANOVA) 
was  used  to  compare  the  two  groups  on  the  basis  of:  (1)  current  enlisted  pay  grade, 
(2)  months  of  maintenance  experience  on  the  NSSMS,  and  (3)  number  of  enlisted 
occupational  specialties  held. 

Results  showed  that  the  two  groups  differed  significantly  (£  <  .01  in  all  cases)  on  the 
above  variables:  The  experienced  technician?  had  a  higher  average  enlisted  pay  grade  (5.7 
versus  3.6),  more  months  of  NSSMS  maintenance  experience  (41.7  versus  4.0),  and  held 
more  enlisted  occupational  specialities  (1.6  versus  0.3)  than  the  inexperienced  group. 
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Performance  Analyses 


Since  the  first  8  dependent  measures  were  categorized  as  troubleshooting  measures 
and  the  remaining  6  as  information  location  measures,  two  separate  analyses  were 
performed,  one  for  each  category. 

Because  8  of  the  28  participants  failed  to  solve  one  of  the  two  troubleshooting 
problems  administered  when  using  NSSMS  technical  manuals,  no  data  were  collected  from 
those  8  with  respect  to  the  information  location  measures.  For  each  of  the  8,  the 
troubleshooting  session  was  terminated  due  to  the  time  criterion  established  by  subject 
matter  experts.  (A  1-hour  time  limit  was  allowed  for  isolating  the  failure  to  the  printed 
circuit  card  level.  If  this  criterion  was  met,  an  additional  15-minute  period  was  allowed 
to  isolate  to  the  component  level;  otherwise  the  session  was  terminated.)  Thus,  the  results 
are  presented  in  two  ways--the  8  troubleshooting  measures  for  all  28  participants  and  the 
troubleshooting  and  information  location  measures  for  the  reduced  sample  of  20  partici¬ 
pants. 

Overall  Sample 

As  shown  in  Table  2,  a  statistically  significant  multivariate  F-ratio  was  obtained  for 
the  presentation  method/troubleshooting  problem  interaction  (H8,  41)  =  4.80,  £  <  0.01). 
Based  on  this  interaction,  MANOVAs  were  computed  to  determine  the  effect  of  the 
presentation  method  on  each  troubleshooting  problem.  Results  from  these  analyses 
showed  a  significant  multivariate  F-ratio  for  the  firing  officer  console  problem  (F(8,  19)  = 
9.71,  £  <  .01),  but  a  nonsignificant  F-ratio  for  the  director  control  problem  (F_(8,  19)  .- 
1.53,  £  <  0.22).  This  finding  led  to  an  examination  of  the  univariate  F-ratios  to  identify 
the  specific  variables  on  which  the  two  presentation  methods  differed  with  respect  to  the 
firing  officer  console  problem.  Table  3  provides  a  detailed  breakdown  of  these 
performance  differences. 


Table  2 

MANOVA  Results  for  Overall  Sample 
(N  s  56  )a 


Part  I:  Multivariate  tests  with  8,  41  degrees  of  freedom  (df). 


Unit  of  Analysis 

F 

£ 

Experience  x  Mode  x.  Problem 

1.09 

.39 

Mode  x  Problem 

4.80 

.01 

Experience  x  Problem 

0.48 

.86 

Experience  x  Mode 

0.95 

.48 

Problem 

9.10 

.0! 

Mode 

5.51 

.01 

Experience 

0.89 

.53 

a5ince  the  NSSMS  troubleshooting  problems  were  treated  as 
an  independent  variable  in  this  MANOVA  design,  56  obser¬ 
vations  were  included  in  the  analysis  (i.e.,  ?§  participants  x 
2  problems  each). 
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Table  3 


Performance  Comparisons  for  the  Overall  Sample 
Concerning  the  Firing  Officer  Console  Problem 
(N  =  2S) 


Measure 

M 

Technical 

Manual 

M 

Automated 

Presentation 

F(l,  26) 

£ 

Time-to-condition  (mins.) 

58.5 

34.8 

38.59 

.01 

Problems  solved  (%) 

53.0 

100.0 

10.56 

.01 

Test  points  checked 

9.1 

8.3 

0.42 

.63 

False  replacements 

0.8 

0.0 

14.04 

.01 

Out-of-bounds  errors 

6.3 

0.4 

28.48 

.01 

Valid  checks 

8.7 

8.2 

0.09 

.77 

Invalid  checks 

1 .9 

0.08 

12.43 

.01 

Proof  points 

0.9 

2.0 

19.82 

.01 

Significant  differences  were  found  between  the  two  technical  information  delivery 
methods  with  respect  to:  (1)  the  overall  time  to  problem  completion,  (2)  the  number  of 
false  replacements  recommended  to  correct  the  failure,  and  (3)  success  in  isolating  the 
problem  to  the  faulty  printed  circuit  card;  thereby  supporting  the  predictions  of  the  first 
and  second  experimental  hypotheses,  respectively.  Although  no  difference  was  found 
between  the  two  presentation  methods  regarding  the  total  number  of  fault  isolation  tests 
performed,  significant  differences  were  found  for  all  but  one  of  the  remaining  dependent 
variables  used  to  assess  the  validity  of  those  tests.  That  is,  technicians  who  used  TMs  for 
the  NSSMS  made  significantly  more  errors  (i.e.,  out-of-bounds  and  invalid  checks)  and 
checked  significantly  fewer  proof  points  than  those  using  the  automated  presentation 
system.  This  finding  supports  the  third  hypothesis. 

Contrary  to  the  predictions  concerning  the  participants'  experience  level  and 
presentation  method  (i.e.,  hypotheses  4  and  5),  no  significant  interaction  was  found 
between  these  factors  for  the  8  troubleshooting  measures  (Table  2).  There  was  a 
significant  interaction,  however,  between  experience  and  presentation  method  for  the  6 
information  location  measures,  which  is  described  in  the  next  section. 


Reduced  Sample 


Results  from  the  20  participants  who  were  successful  were  analyzed  separately.  The 
results  from  the  multivariate  analysis  of  7  of  the  8  troubleshooting  measures  for  the 
reduced  sample  mirrored  the  previous  analysis.  (The  dichotomous  "pass"  or  "fad"  score 
was  excluded  from  the  latter  analysis  since  this  measure  had  no  variance.)  There  was  a 
significant  interaction  between  troubleshooting  problem  and  presentation  method.  As  in 
the  analysis  for  the  overall  sample,  no  difference  between  the  two  presentation  methods 
with  respect  to  the  director  control  unit  problem  was  found.  Accordingly,  Table  4 
presents  the  performance  differences  attributed  to  the  presentation  method  factor  for  the 
firing  officer  console  problem. 


Table  4 


Performance  Comparisons  for  the  Reduced  Sample 
Concerning  the  Firing  Officer  Console  Problem 
(N  =  20) 


Measure 

M 

Technical 

Manual 

M 

Automated 

Presentation 

F(l,  19) 

£ 

Time-to-condition  (mins.) 

56.6 

34.8 

18.10 

.01 

Test  points  checked 

10.6 

8.3 

2.42 

.14 

False  replacements 

0.8 

0.0 

15.12 

.01 

Out-of-bounds  errors 

4.6 

0.4 

26.88 

.01 

Valid  checks 

10.5 

8.2 

1.97 

.18 

Invalid  checks 

2.4 

0.08 

20.03 

.01 

Proof  points 

1 .6 

2.0 

7.06 

.05 

With  regard  to  the  remaining  6  dependent  measures  (the  information  location 
measures),  MANOVA  results  (summarized  in  Table  5)  showed  significant  interactions  for 
experience  level  and  presentation  method  (Fj6,  35)  =  3.25,  £  <  .05)  and  for  the 
troubleshooting  problem  and  presentation  method  (F(6,  35)  =  2.55,  £  <  .05).  For  each  of 
these  interactions,  only  a  single  dependent  variable  differed  reliably  at  the  univariate 
level  of  analysis:  success  in  locating  narrative  descriptions  of  the  faulty  circuitry  for  the 
first  interaction  effect;  time  to  locate  parts  information  for  the  second.  More 
specifically,  results  showed  that:  (1)  all  technicians  who  used  the  automated  presentation 
system  were  successful  in  locating  appropriate  narrative  information,  compared  to  71 
percent  of  the  experienced  and  33  percent  of  the  inexperienced  technicians  who  used  TMs, 
and  (2)  technicians  who  used  the  automated  presentation  system  took  significantly  less 
time  to  locate  parts  information  for  both  the  firing  officer  console  and  director  problems 
(M  =  1.07  and  1.13  minutes,  respectively),  compared  with  technicians  who  used  TMs  (N1  r 
4.9  and  8.7  minutes  for  the  firing  officer  console  and  director  problems,  respectively). 

User  Evaluation  Questionnaire 

The  results  from  the  25  items  in  the  User  Evaluation  Questionnaire  (Appendix  C)  are 
shown  in  Table  6.  Each  question  had  a  5-point  scale  ranging  from  1  (unsatisfactory  or 
significantly  less)  to  5  (outstanding  or  significantly  more).  Overall,  the  NSSMS 
technicians  rated  the  automated  presentation  system  from  3  (satisfactory)  to  4  (highly 
satisfactory).  Although  various  physical  characteristics  were  assessed  in  the  first  10 
questionnaire  items,  a  single  mean  (3.59)  is  shown  for  these  items  to  reflect  the  similarity 
in  the  ratings  assigned  to  those  characteristics.  Similarly,  the  technicians  rated  the 
adequacy  of  the  computer  operation  and  software  features  in  the  "satisfactory"  to  "highly 
satisfactory"  range  =  3.74).  Use  of  the  automated  system  was  also  perceived  by  the 
technicians  as  an  improvement  over  TMs  for  the  NSSMS  in  terms  of  efficiency  and 
effectiveness  in  providing  needed  maintenance  information.  In  addition,  no  differences 
were  found  between  the  ratings  of  experienced  and  inexperienced  technicians  relative  to 
the  four  main  categories  of  questionnaire  items. 
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Table  5 


MAN'OVA  Results  for  Reduced  Sample  on 
Supplemental  Performance  Measures 
(N  =  48)a 


Part  I:  Multivariate  tests  with  5,  35  degrees  of  freedom  (df). 


Unit  of  Analysis 

F 

£ 

Experience  x  Mode  x  Problem 

2.02 

.09 

Mode  x  Problem 

3.24 

.05 

Experience  x  Problem 

0.56 

.76 

Experience  x  Mode 

2.55 

.05 

Problem 

3.51 

.01 

Mode 

22.87 

.01 

Experience 

2.38 

.05 

o 

Supplementary  performance  data  were  not  obtained  for  the 
S  technicians  who  failed  to  solve  the  NSSMS  troubleshoot¬ 
ing  problems  when  using  technical  manuals  (TMs). 


Table  6 

Summarv  of  User  Evaluation  Questionnaire 
(N  =  2S) 


Feature 

Questionnaire 

Items 

M 

Physical  features 

1-10 

3.59 

Operation  &  software  features 

11-17 

3.74 

Accessing  information  problems 

18-20 

2.48 

Effectiveness  of  information 

21-25 

3.97 

Scale  Values: 

Questions  1-17:  1  =  unsatisfactory;  5  =  outstanding. 

Questions  18-25:  1  =  significantly  less;  5  =  significantly  more. 

(Low  values  for  Questions  18-20  reflect  a  positive  response.) 

A  review  of  the  open-ended  comments  found  that  the  technicians'  overall  impression 
of  the  electronic  delivery  system  was  generally  favorable.  The  primary  criticism 
concerned  the  size  of  the  display  screen  for  presenting  schematic-type  diagrams,  that  is, 
the  diagrams  were  too  small,  thus  precluding  a  view  of  them  in  their  entirety. 


15 


% 


V*, 

A 


PART  D:  IMPLEMENTATION  ISSUES 


The  technical  objective  of  Part  II  was  to  obtain  information  related  to  fleet 
implementation  of  an  automated  presentation  system,  such  as  PEAM.  The  implementa¬ 
tion  issues  addressed  included:  (1)  assessing  the  user's  (the  NSSMS  technician)  needs  for  an 
automated  presentation  system,  (2)  identifying  logistical  support  considerations, 
(3)  determining  training  implications,  and  (4)  identifying  interfaces  with  other  automated 
shipboard  data  bases.  As  noted  in  Part  I,  hardware  and  software  problems  precluded  the 
use  of  an  operable  PEAM  device.  Thus,  data  collected  during  this  part  of  the  evaluation 
were  based  more  on  a  conceptual  narrative  description  of  an  automated  presentation 
system  than  on  an  operable  prototype. 

Participants 

Information  was  collected  from  two  different  groups  of  personnel.  One  group 
consisted  of  25  enlisted  NSSMS  technicians  who  completed  a  structured  questionnaire 
(shown  in  Appendix  D).  Each  technician  in  the  sample  had  a  minimum  of  6  months’  fleet 
experience  in  troubleshooting  NSSMS.  Fourteen  technicians  were  from  SPRUANCE-class 
destroyers  (DD  963)  and  11  were  from  aircraft  carriers  (CV).  The  mean  time  these 
technicians  had  served  as  fire  control  technicians  was  3.2  years,  with  a  range  from  1  to  9 
years.  The  mean  time  they  served  as  NSSMS  technicians  was  1.5  years,  with  a  range  from 
S  months  to  7  years. 

There  were  nine  in  the  second  group:  seven  NSSMS  "C"  school  instructors,  on  NSSMS 
documentation  expert,  and  the  project  director  for  the  development  of  an  automated  data 
base  for  the  MK  15  (PHALANX)  Close-In  Weapon  System. 

Method 

Technicians  in  the  first  group  were  told  that  the  Navy  and  the  other  services  were 
investigating  the  feasibility  of  transitioning  from  paper-based  TMs  to  an  automated 
presentation  system  for  maintenance  information.  PEAM  was  described  as  one  example 
of  how  the  automated  presentation  system  concept  might  be  accomplished.  During  the 
presentation  of  the  PEAM  concept,  the  technicians  had  an  opportunity  to  ask  questions. 
They  then  completed  the  questionnaire. 

For  the  second  group,  open-ended  interview  questions  were  used.  An  orientation  was 
given  about  the  possible  transition  from  paper  TMs  to  an  automated  presentation  system. 
Each  participant  was  asked  to  reflect  on  the  work  they  were  doing  and  how  this  transition 
might  impact  on  that  work.  Specifically,  the  "C"  school  instructors  were  asked  about  the 
impact  on  the  training  of  NSSMS  technicians;  the  NSSMS  documentation  expert  was  asked 
to  consider  how  the  implementation  of  a  device  such  as  a  PEAM  could  impact  his  work  in 
the  development  of  technical  documentation;  and  the  project  director  was  asked  to  talk 
about  his  experience  related  to  the  development  of  an  automated  data  base. 

Results 

The  questionnaire  data  from  the  25  technicians  are  given  in  Appendix  D.  A  summary 
of  these  data  along  with  the  information  from  the  open-ended  interview  questions 
pertinent  to  each  area  of  consideration  is  provided  below. 
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User  Considerations 


All  but  one  technician  thought  that  two  or  three  PEAM  devices  would  be  needed  t- 
maintain  a  single  NSSMS  system.  The  rationale  was  that  since  NSSMS  consists  of  multi 
components  distributed  throughout  the  ship,  it  would  be  convenient  to  have  a  PEAM 
device  placed  at  more  than  one  decentralized  location,  as  simultaneous  maintenance 
activities  are  sometimes  required  on  NSSMS  components,  and  multiple  devices  per  NSSMS 
would  provide  backup  capability  if  one  became  inoperative.  Most  of  the  technicians  also 
preferred  that  the  PEAM  memory  modules  be  co-located  with  each  PEAM  if  more  than 
one  set  of  modules  was  provided.  If,  on  the  other  hand,  only  one  set  of  memory  modules 
was  provided  per  NSSMS,  most  of  the  technicians  preferred  decentralized  storage 
locations  for  the  PEAM  devices  v,ith  the  memory  modules  distributed  to  each  device 
location.  The  primary  reasons  were  convenience  and  survivability. 

The  majority  of  technicians  preferred  a  functional  organization  of  the  maintenance 
information  in  the  automated  presentation  system.  This  would  match  the  current 
organization  of  the  TMs,  which  eliminates  the  need  to  search  for  each  unit.  The 
functional  approach  "••as  also  preferred  for  labeling  the  memory  modules. 

The  project  director  indicated  that  a  key  task  will  be  to  design  an  index  or  indices 
that  will  allow  the  technician  to  access  information  as  quickly  as  possible.  Most  of  the 
technicians  preferred  a  table  of  contents-type  of  indexing  located  on  one  memory 
module. 

One  of  the  NSSMS  instructors  suggested  that  PEAM  have  a  marking  system  so  that  a 
technician  could  "mark"  the  place  being  used  if  called  away  to  perform  a  collateral  duty. 
It  would  then  be  possible  for  the  technician  quickly  return  to  the  information  accessed 
previously,  even  though  another  technic  in  may  have  used  the  same  module  in  the  interim. 


Logistical  Support  Considerations 

The  technicians  were  evenly  divided  on  where  updating  and  correcting  of  PEAM 
memory  modules  should  take  place.  Reasons  supporting  shipboard  updating  included 
efficiency,  accountability,  and  the  fact  that  the  technician  would  be  aware  of  what 
information  needed  revision.  Updating  at  shore-base  commands,  on  the  other  hand,  is 
better  for  quality  control  and  standardization.  It  was  also  mentioned  that  fleet 
technicians  might  inadvertently  erase  or  damage  the  contents  of  the  modules  durinfc  the 
updating  or  correction  process. 

Because  of  potential  damage  to  or  loss  of  memory  modules  from  a  variety  of  causes, 
most  of  the  technicians  preferred  a  shipboard  capacity  for  producing  backup  copies.  This 
arrangement  was  thought  to  be  more  efficient  and  would  also  save  paperwork.  About  half 
the  technicians  thought  that  damage  or  degradation  of  the  modules  could  occur  from 
transmitter  emissions  or  when  PEAM  was  used  near  the  NSSMS  launcher  or  director. 
Damage  to  the  PEAM  device  and  memory  modules  might  also  result  from  exposure  to 
water,  high  temperatures,  dust,  and  other  airborne  particulate  matter. 

The  NSSMS  documentation  expert  stated  that  the  development  of  a  technical 
information  data  base  involves  a  variety  of  participants  in  both  the  government  and 
private  sectors.  U'ith  current  paper  TMs,  there  is  the  time-consuming  process  of  sending 
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examples  and  copies  of  materials  from  the  TM  originator  to  the  approving  agency  and  vice 
versa.  The  documentation  expert  felt  that  the  use  of  an  automated  system  would  render 
the  development  process  more  efficient.  All  participants  could  view  the  same  informa¬ 
tion  simultaneously  and  make  any  necessary  additions  and  corrections  instantaneously. 
The  updating  process  will  also  be  more  efficient  for  both  the  agency  responsible  for 
making  changes  and  for  the  technician.  Currently,  the  revised  pages  are  mailed  to  the 
appropriate  work  center  aboard  ship  where  the  technical  manuals  are  stored.  A 
technician  in  the  work  center  must  then  remove  the  affected  pages  from  the  TM  and 
insert  the  pages  with  the  updated  information.  With  the  use  of  an  automated  system,  the 
technician  would  most  likely  update  the  technical  information  system  electronically,  by 
means  of  a  magnetic  tape,  diskette,  or  optical  disc  containing  all  the  relevant  changes. 

The  technicians  preferred  that  they  be  responsible  for  maintenance  of  the  PEAM 
device.  They  also  preferred  that  the  maintenance  information  for  the  automated 
presentation  system  itself  take  the  form  of  planned  maintenance  system  cards  or  a  paper 
TM. 

Most  of  the  technicians  cited  situations  and/or  conditions  that  would  require  a  paper 
copy  of  information  contained  in  PEAM  memory  modules.  For  example,  two  or  more 
technicians  could  produce  paper  copies  of  selected  portions  of  a  memory  module,  then 
take  that  information  to  various  equipment  sites  aboard  ship.  The  NSSMS  computer  room 
was  the  preferred  shipboard  location  for  a  printer.  While  a  printer  was  preferred,  more 
than  half  the  technicians  perceived  problems,  such  as  having  responsibility  for  maintaining 
the  printer  and  ordering  supplies  for  it. 


Training  Considerations 


It  was  felt  that  training  in  the  use  of  PEAM  would  most  likely  occur  either  in  the 
technical  training  schools  or  in  the  fleet.  Most  of  the  technicians  indicated  that  the 
extant  NSSMS  "C"  school  would  be  the  best  place  for  preliminary  training  in  the  operation 
of  an  automated  presentation  system.  For  those  technicians  who  do  not  receive  training 
on  PEAM  in  "C"  school,  an  on  line  demonstration  and  instruction  book  should  be  made 
available  aboard  ship. 


The  importance  of  shipboard  training  was  emphasized  by  one  of  the  NSSMS  "C"  school 
instructors  who  ..oted  that  when  new  technicians  report  aboard  ship,  they  very  often  do 
not  have  an  opportunity  to  engage  in  troubleshooting  for  6  to  12  months.  When  they  do 
begin  to  troubleshoot,  much  of  the  information  lear  ed  concerning  the  organization  of  the 
TMs  has  to  be  relearned.  A  TM  refresher  course  could  be  built  into  the  automated 
presentation  system. 

Responses  from  the  "C"  school  instructors  suggest  that  NSSMS  maintenance  training 
courses  could  be  enhanced  with  feedback  from  the  fleet  on  troubleshooting  experiences. 
For  example,  the  automated  presentation  system  (PEAM)  could  be  programmed  to  capture 
the  troubleshooting  strategies  used  by  NSSMS  technicians.  This  information  might  include 
such  factors  as:  symptom  recognition  and  elaboration,  amount  and  type  of  technical 
information  accessed,  test  points  checked,  and  proposed  solutions.  From  these  data, 
scenarios  could  be  developed  into  case  studies  for  use  by  the  technical  training  schools. 


Although  a  variety  of  training  applications  were  identified  for  PEAM,  there  was  little 
consensus  among  the  technicians  as  to  any  one  specific  application.  Nevertheless,  it  was 
felt  that  PEAM  could  be  useful  for  providing  training  in  the  areas  of  general  damage 
control,  general  maintenance,  and  the  use  of  fire  fighting  equipment. 
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Interface  Considerations 

The  technicians  thought  that  PEAM  could  be  expanded  for  use  as  an  automated  fault 
diagnostic  aid.  This  could  take  the  form  of  a  stand-alone  device  to  which  equipment 
casualty  symptoms  would  be  inputted  for  resolution  through  its  expert  reasoning 
capability.  Alternatively,  an  automated  presentation  system  with  diagnostic  capability 
could  interface  with  built-in  automatic  test  equipment  currently  used  to  support  N5SMS 
diagnostics.  Suggestions  were  also  made  for  providing  the  automated  presentation  system 
with  a  capability  to  automate  certain  shipboard  maintenance  support  functions  that  are 
currently  performed  manually,  such  as  preparing  equipment  casualty  reports  and  docu¬ 
menting  planned  or  completed  preventive  maintenance  activities. 


DISCUSSION 

The  results  of  this  comparative  assessment  of  paper-based  and  computer-based 
maintenance  information  delivery  systems  are  discussed  in  terms  of  (1)  effect  on 
performance,  (2)  implementation  issues,  and  (3)  impact  of  innovative  changes  on  the 
Navy's  personnel  subsystem. 

Effect  on  Performance 

Results  from  this  study  showed  that  the  use  of  electronically  delivered  technical 
information  during  troubleshooting  was  more  effective  than  the  use  of  paper-based  TMs. 
The  degree  of  effectiveness,  however,  was  related  to  troubleshooting  complexity.  While 
it  had  been  the  initial  intent  to  use  two  troubleshooting  problems  of  comparable 
difficulty,  it  became  apparent  in  the  analyses  of  the  resultant  performance  data  that  the 
two  problems  were,  indeed,  different.  The  firing  officer  console  problem  represented  a 
considerably  more  difficult  troubleshooting  task  than  the  director  control  unit  problem. 
As  such,  no  significant  differences  in  performance  were  observed  for  the  director  control 
unit  problem  regardless  of  presentation  method.  Thus,  the  payoff  in  effectiveness  for 
electronically  delivered  technical  information  appears  to  be  in  accomplishing  the  more 
complex  troubleshooting  tasks,  such  as  those  represented  by  the  firing  officer  console 
problem. 

When  technicians  were  asked  to  troubleshoot  the  firing  officer  console  problem,  use 
of  the  automated  presentation  system  resulted  in  a  69  percent  reduction  in  total 
troubleshooting  time.  More  importantly,  there  was  a  100  percent  success  rate  in  fault 
isolation  compared  with  a  53  percent  rate  for  those  using  the  paper-based  TMs.  (There 
was  a  100%  success  rate  with  the  automated  presentation  system  for  the  director  control 
problem  as  well,  although  the  success  rate  using  TMs  was  also  high,  92%. )  There  were  no 
false  replacements  in  troubleshooting  the  firing  olficer  console  when  using  the  automated 
presentation  system,  while  the  average  when  using  TMs  was  slightly  less  than  one  false 
replacement  (0.70).  Other  error  measure  comparisons  for  the  firing  officer  console 
problem  also  supported  the  effectiveness  of  the  automated  presentation  system:  Tech¬ 
nicians  checked  twice  as  many  proof  points  and  were  12  times  less  likely  to  commit  an 
out-of-bounds  error. 

These  data  suggest  that  savings  of  more  than  one  kind  are  possible.  For  example, 
current  field  data  for  the  NSSMS  indicate  that  the  mean-time-to-repair  (MTTR)  for  the 
firing  officer  console  is  8.1  hours.  The  40  percent  reduction  in  troubleshooting  time 
obtained  from  the  present  study  would  reduce  the  MTTR  for  the  firing  officer  console  to 
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approximately  4.9  hours.  Field  data  also  provide  a  measure  related  to  false  replacements, 
viz,  no  evidence  of  failure  (NEOF).  The  firing  officer  console  NEOF  rate  for  a  12-month 
period  in  19$6  was  9.  This  value  is  close  to  the  1 1  false  replacements  made  by  the 
participants  who  used  the  paper  TMs  in  the  present  study.  Switching  to  an  automated 
presentation  system  has  the  potential  for  reducing  the  firing  officer  console  NEOF  rate  to 
zero. 

The  automated  presentation  system  also  dramatically  improved  the  technicians' 
ability  to  reference  only  the  information  needed  to  isolate  equipment  faults.  When  the 
technicians  used  the  automated  presentation  system  they  performed  fewer  tests  (a  ratio 
of  1:1.3)  and  had  fewer  invalid  checks  (a  ratio  of  1:3)  compared  with  when  they  used  the 
TM.  By  providing  a  more  efficient  way  of  accessing  technical  information,  the  automated 
presentation  system  should  help  maintenance  technicians  become  more  effective  in  less 
time  when  isolating  equipment  faults. 

Although  there  was  an  expectation  that  the  maintenance  experience  of  the  tech¬ 
nicians  would  have  a  differential  effect  across  all  dependent  measures,  there  was  only  a 
single  measure  in  which  there  was  a  difference  between  inexperienced  and  experienced 
technicians.  Specifically,  there  was  a  difference  only  when  the  participants  were  asked  to 
locate  narrative  information  for  the  two  faulty  printed  circuit  cards.  When  using  the 
TMs,  the  experienced  group  had  a  higher  success  rate  (71%)  in  locating  the  appropriate 
narrative  information  than  the  inexperienced  group  (33%).  This  finding  is  in  sharp 
contrast  to  the  100  percent  success  rate  fcr  both  the  experienced  and  inexperienced 
technicians  who  used  the  automated  presentation  system.  Thus,  it  appears  that  the 
automated  presentation  system  offers  a  distinct  advantage  for  inexperienced  technicians 
in  locating  supplementary  maintenance  information. 

Results  from  the  questionnaire  indicate  that  the  technicians  favor  an  automated 
presentation  system  for  accessing  NSSMS  technical  information.  In  addition,  the 
technicians  pointed  out  specific  areas  that  they  believed  improved  performance,  and  areas 
they  felt  needed  to  be  improved.  Examples  of  favorable  comments  were:  "Information 
was  available  faster,  at  the  press  of  a  button,"  and  "1  did  not  hive  to  flip  througn  pages 
and  change  from  one  paper  volume  to  another,  since  branching  to  another  'frarne/p.’.ge'  of 
information  with  the  computer  was  automatic." 

Since  screen  size  was  limited,  technicians  had  some  difficulty  relating  to  signal  flow 
information  that  had  to  be  sectioned  in  accordance  with  the  physical  size  of  the  computer 
f-ame.  The  technicians  were  used  to  viewing  11  x  17-inch  pages  in  TMs  for  that  type  of 
information.  With  the  5  x  7-inch  display  area  of  the  automated  presentation  system,  the 
technicians  experienced  some  difficulty  in  maintaining  an  overall  perspective  of  how  the 
circuit  components  were  linked  together.  Therefore,  the  technicians  suggested  that  a 
windowing  or  scrolling  capability  be  provided  to  allow  concurrent  viewing  of  signal  inputs 
and  outputs  located  on  adjacent  pages/frames.  Such  comments  are  interesting  in  that  the 
technicians  appear  to  recognize  that  an  exact  electronic  duplicate  of  the  11  x  17-inch 
page  from  the  TM  would  not  be  required.  Windowing  could  also  be  used  allow  viewing 
the  signal  flow  diagram  of  interest  while  opening  a  window  for  other  technical  informa¬ 
tion,  such  as  narrative  text  or  other  test  points. 

The  performance  data  and  the  questionnaire  responses  from  the  present  study  are 
also  supported  by  findings  conducted  by  the  Air  Force,  Army,  and  Navy. 


In  an  Air  Force  study  (Clay,  1986),  a  prototype  computer-based  maintenance  aid 
system  (CMAS)  was  developed  to  evaluate  technical  data  presentation  and  user  interface 
techniques.  An  off-the-shelf  computer  was  used  as  the  prototype  delivery  device. 
Software  was  developed  to  store  and  present  technical  information  in  an  integrated  text- 
graphics  format.  The  technical  information  supported  the  RT-728A/APX-64(V)  radio 
receiver  transmitter.  An  evaluation  of  CMAS  was  accomplished  by  having  technicians 
perform  adjustment  and  alignment  tasks  using  the  technical  information  presented  via 
CMAS.  Eight  technicians  completed  the  assigned  maintenance  tasks  and  data  were 
collected  on  the  technicians'  task  times  and  attitudes  toward  CMAS.  In  general,  the 
technicians  expressed  positive  comments  about  the  use  of  CMAS  to  support  maintenance. 
Since  the  primary  purpose  of  the  CMAS  study  was  only  to  demonstrate  the  concept  of 
electronic  delivery  of  technical  information,  only  limited  data  comparing  fault  isolation 
using  CMAS  with  the  conventional  paper  technical  manual  were  collected. 

In  a  follow-on  to  CMAS,  the  Navy  (Nugent,  Sander,  Johnson,  &  Smillie,  1987) 
compared  troubleshooting  performance  using  electronically  presented  technical  informa¬ 
tion  with  that  using  standard  TMs.  The  technical  information  was  for  the  same  equipment 
as  the  Air  Force  study,  the  RT-728A/APX-64(V)  radio  receiver  transmitter.  Thirty-six 
technicians  in  the  Navy,  Marines,  and  Air  Force  participated  and  attempted  to  isolate  two 
printed  circuit  card  failures  using  the  technical  information  contained  in  an  automated 
presentation  system  and  two  other  failures  when  using  paper  TMs.  All  faults  were  found 
when  the  automated  system  was  used,  whereas  there  were  nine  occurrences  where 
technicians  failed  to  isolate  one  or  both  of  the  faults  when  using  the  TMs.  With  the 
automated  presentation,  technicians  isolated  faults  in  about  one-half  the  time  and  made 
no  false  replacements. 

In  two  Navy  Technical  Information  Presentation  Program  (NTIPP)  studies,  main¬ 
tenance  performance  was  compared  when  technicians  used  either  the  standard  paper  TMs, 
reformatted  paper  TMs,  or  an  automated  presentation  system.  In  the  first  NTIPP  study, 
which  involved  the  F-14A  Rudder  Control  System,  2k  technicians  used  one  of  the  three 
formats  for  non-troubleshooting  tasks  (e.g.,  remove/replace)  and  either  the  standard  paper 
TMs  or  the  automated  system  for  a  troubleshooting  task  (Fuller,  Post,  &  Mavor,  1987). 
Employing  a  somewhat  unbalanced  design,  the  authors  found  no  apparent  performance 
differences  among  the  Three  formats  for  supporting  non-troubleshooting  tasks.  There  was, 
however,  an  improvement  in  success  rate  for  troubleshooting  tasks  when  using  an 
automated  system.  All  technicians  who  used  the  automated  system  found  the  fault, 
compared  with  seven  technicians  who  failed  to  find  it  when  using  the  standard  paper  TMs. 
Regardless  of  the  performance  data  outcome,  all  but  one  of  the  participants  favored 
automatic  presentation  over  the  paper  TM. 

A  similar  design  was  employed  in  the  second  NTIPP  study  (Fuller,  LeBeau,  Mavor, 
Post,  &  Sawyer,  1987).  Twenty-four  technicians  were  tested  using  technical  information 
that  supported  the  SPA-25D  radar  system.  Results  suggested  that  troubleshooting 
performance  time  can  be  reduced  by  approximately  24  percent  when  using  an  automated 
system.  In  addition,  the  majority  of  the  technicians  (22  cf  the  24)  preferred  the 
automated  system  overall,  found  the  automated  presentation  easier  to  use,  and  said  the 
data  base  for  the  automated  system  was  better  organized. 

In  an  Army  study  (Department  of  the  Army,  1987),  the  use  of  an  interactive  video 
disc  was  compared  with  that  of  paper  TMs  to  perform  maintenance  tasks.  Ten  fwo-person 
operator-maintainer  teams  performed  302  tasks  (i.e.,  fault  isolation,  remove  and  install, 
repair  and  verify,  preventive  maintenance,  operation,  and  repair  parts  identification)  on 
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the  PATRIOT  Missile  System.  Half  used  the  interactive  video  and  half  used  the  paper 
TMs.  Although  there  were  no  differences  in  performance  between  the  two  presentation 
methods,  all  but  one  technician  favored  the  electronic  presentation  over  the  paper  TM. 

Taken  together,  results  from  the  foregoing  studies  demonstrate  the  potential  benefits 
of  automating  the  presentation  of  technical  information.  In  general,  data  from  these 
studies  along  with  data  from  the  present  study  show  how  maintenance  performance  might 
be  improved  with  respect  to  such  factors  as  success  in  fault  isolation,  time  to  complete 
maintenance  tasks,  and  reduced  error  rates. 

Implementation  Issues 

Although  the  present  study  was  oriented  primarily  toward  a  specific  application  of  an 
automated  technical  information  presentation  system,  results  from  Part  II  of  the  present 
effort  highlighted  concerns  that  must  be  addressed  prior  to  fleet  implementation  of  a 
PEAM-like  device. 

To  ensure  adequate  coverage  and  availability,  multiple  delivery  devices  and  several 
copies  of  the  data  base  are  necessary.  Equally  important  is  distribution  of  the  delivery 
devices  throughout  the  platform  the  weapons  support.  Backup  copies  of  the  automated 
technical  information  data  base  are  a  necessity  for  shipboard  applications.  It  also  seems 
worthwhile  to  ensure  compatibility  between  other  computer-based  delivery  devices  and 
software  so  that  technical  information  designed  for  one  device  can  be  readily  accessed 
from  another,  if  necessary. 

Transition  from  a  paper  to  an  electronic  medium  for  authoring  and  storing  technical 
information  should  make  the  development  cycle  for  technical  information  more  efficient, 
in  particular  the  review  and  evaluation  phases.  In  addition,  revisions  and  updates  should 
be  faster  and  easier  to  incorporate  and  distribute,  most  likely  as  completely  revised 
"manual  sets."  Thus,  the  time-consuming,  error-prone  task  of  page  replacement/deletion 
will  be  eliminated. 

Two  issues  must  be  addressed  relative  to  personnel  training.  First,  there  is  a  need  for 
training  in  the  operation  of  the  automated  presentation  system  itself.  While  the 
necessary  training  should  be  simple  and  straightforward,  the  training  should  be  integrated 
into  the  system  training  courses  at  the  technical  schools.  Second,  the  automated 
presentation  system  can  serve  as  a  vehicle  for  both  on-the-job  training  and  participatory 
shipboa.d  training.  As  such,  technicians  could  use  the  automated  presentation  system  to 
acquire  new  skills  and  knowledge  related  to  the  particular  weapon  system  or  to  access 
othet  types  of  training  (e.g.,  damage  control,  the  planned  maintenance  system,  personnel 
qualification  standards). 

The  power  and  advantages  of  an  automated  presentation  system  will  not  be  fully 
realized  if  the  system  is  restricted  to  a  single  interface,  that  is,  between  the  technician 
and  the  technical  information  data  base.  Since  most  advanced  weapon  systems  in  the 
Navy  contain  built-in  test  (BIT)  capabilities,  consideration  should  be  given  to  developing 
an  interface  that  would  allow  the  automated  presentation  system  to  initiate  BIT  routines 
and  process  BIT  data  directly.  Such  an  interface  would  eliminate  the  need  for  the 
technician  to  initiate  BIT  and  then  input  the  test  results  for  subsequent  evaluation  by  the 
system.  An  automated  system  equipped  with  an  "expert  reasoning"  capability  could 
evaluate  the  BIT  results  and  provide  the  technician  with  a  menu  of  alternative,  most 
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likely,  next  actions  to  perform;  thereby  minimizing  the  role  of  the  technician  as  a  data 
transfer  agent  and  maximizing  that  person  as  an  analytic  decision  maker.  Interfaces  with 
other  shipboard  data  bases  should  be  considered,  such  as  the  shipboard  non-tactical  ADP 
program  (SNAP),  the  consolidated  automatic  support  system  (CASS),  etc.,  to  provide 
maximum  usage  of  available  maintenance  data  and  to  simplify  parts  ordering  and 
documentation  of  maintenance  actions. 

Innovation  Changes 

As  with  any  innovation,  personnel  issues  must  be  considered.  The  primary  emphasis 
in  the  design  and  development  of  any  technical  information  delivery  system  should  focus 
on  the  end  user,  in  this  case,  a  Navy  maintenance  technician. 

With  paper  TMs,  technicians  have  considerable  latitude  to  formulate  their  own  fault 
isolation  strategy  based  on  the  symptoms  of  the  fault.  Thus,  an  experienced  technician's 
repository  of  maintenance  skill  and  knowledge  may  be  based  on  the  way  in  which 
information  is  organized  within  the  TM.  Inexperienced  technicians,  on  the  other  hand, 
may  not  be  able  to  quickly  locate  needed  technical  information  from  the  TM.  As  the 
technician  gains  experience,  however,  he  or  she  gradually  develops  more  efficient  ways  of 
accessing  that  information.  With  an  electronic  presentation  system,  there  may  be  the 
tendency  on  behalf  of  designers  to  capitalize  on  efficiency  by  directing  the  users  along  a 
predefined  path,  regardless  of  their  experience.  A  question  arises,  then,  in  terms  of 
direction.  Should  more  reliance  be  placed  on  the  automated  presentation  system  to 
automatically  display  the  most  efficient  and  effective  ways  of  using  technical 
information'!’  Or,  alternatively,  should  the  technician  develop,  through  experience,  the 
skill  and  knowledge  necessary  to  access  all  relevant  information  about  a  maintenance 
problem?  It  appears  that  some  type  of  compromise  must  be  struck  by  the  system 
designers,  one  in  which  the  technician  can  either  follow  a  predefined  path  or  access  the 
information  according  to  persona!  preference  (as  the  technician  gains  experience  and 
knowledge). 

A  related  issue  concerns  supervisors'  perceptions  of  automated  presentation  systems. 
Will  the  use  of  such  a  system  lead  to  higher  expectations  of  efficiency  without  regard  to 
the  human-machine  interface  requirement?  Probably,  the  most  important  issue  is  related 
to  career  advancement,  that  is,  will  there  be  well-defined  career  paths  for  technicians 
using  an  "automated"  device?  or  will  there  be  an  attempt  to  eliminate  the  human-in-the- 
ioop  altogether?  or  will  there  be  a  move  to  settle  for  technicians  with  lesser  skills? 
Complete  reliance  on  an  automated  delivery  system  will  very  likely  result  in  a  complete 
dependence  on  that  system.  Although  the  exact  role  of  the  technician  may  change  in  the 
future  compared  with  that  of  a  technician  whe  has  been  on  the  same  job  for  20  years, 
consideration  must  still  be  given  to  providing  technicians  with  the  capability  to  initiate 
other  fault  isolation  strategies  when  the  automated  approaches  fail.  While  career 
structure  is  primarily  a  social/policy  issue,  maximum  effectiveness  of  any  automated 
technical  information  presentation  system  depends  heavily  on  its  integration  into  the 
Navy's  personnel  subsystem,  heretofore  a  factor  that  has  not  been  adequately  addressed. 
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CONCLUSIONS 


Based  on  the  first  part  of  the  evaluation,  troubleshooting  performance  was  improved 
when  the  automated  presentation  system  was  used  to  display  troubleshooting  information 
for  the  NATO  SEA5PARROW  Surface  Missile  System.  Specifically: 

1.  The  more  difficult  the  problem,  the  more  effective  the  automated  presentation 
system. 

2.  Use  of  the  automated  presentation  system  led  to  significant  reductions  in 
troubleshooting  time  (up  to  40%)  and  error  rates  (fewer  false  replacements  and  fewer 
unnecessary  tests),  while  at  the  same  time  resulting  in  100  percent  success  rate  for  fault 
isolation. 


3.  Regardless  of  experience  level,  technicians  were  more  successful  and  took  less 
time  locating  additional  information  when  using  the  automated  presentation  system. 

k.  Technicians  preferred  an  automated  presentation  system  over  paper-based  TMs. 

Based  on  the  second  part  of  the  evaluation,  the  authors  conclude  that  the  transition 
from  a  paper  'o  electronic  medium  for  presentation  of  technical  information  will  have  a 
major  impact  on  the  Navy's  personnel  subsystem  in  two  important  respects:  (1)  the  way  in 
which  technicians  interact  with  technical  information  contained  in  the  automated  data 
base,  and  (2)  the  changing  role  of  maintenance  technicians  as  they  become  fully  or  even 
partially  supported  by  an  automated  presentation  system.  The  authors  also  conclude  that 
for  shipboard  maintenance  applications,  the  automated  presentation  system  must  be 
available  at  dispersed  locations  to  ensure  access  and  survivability. 

Finally,  it  should  be  noted  that  the  Office  of  the  Secretary  of  Defense  established  the 
Computer  Aided  Logistics  Support  (CALS)  initiative  in  September  1985.  A  major  purpose 
of  CALS  is  to  apply  existing  and  emerging  computer  technology  to  automate  the 
development,  delivery,  and  maintenance  of  logistic  and  technical  information  across  all 
branches  of  the  Armed  Forces.  Among  the  major  objectives  recommended  to  support 
CALS  are  "extensive  study  of  user  interaction  with  non-paper  maintenance  aids"  and 
"research,  experimentation,  and  field  trial  experience  to  find  out  what  forms  of  displays 
are  best  for  specific  situations"  (Riddell,  Cunkel,  Reiser,  Goldstein,  <5c  Lepisto,  1985). 
Although  initiated  prior  to  the  establishment  of  CALS,  the  present  evaluation  does,  in 
large  part,  address  these  objectives. 
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APPENDIX  A 


EXTRACT  FROM  NAVY  TEST  PLAN  FOR  EVALUATING  THE 
PERSONAL  ELECTRONIC  AID  FOR  MAINTENANCE  (PEAM): 
EXECUTIVE  SUMMARY 
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PURPOSE 
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The  primary  purpose  of  this  effort  is  to  compare  the  efficiency  and  effectiveness  of 
an  electronically  delivered  technical  information  system  with  that  of  a  paper-based 
documentation  system  in  supporting  the  information  needs  of  maintenance  technicians. 
This  effort  will  also  address  issues  related  to  fleet  implementation  and  acceptance  of  the 
PEAM  device.  Accordingly,  the  test  plan  consists  of  two  parts:  (1)  performance 
evaluations  based  on  both  preventive  and  corrective  maintenance  activities  and 
(2)  assessment  of  fleet  implementation  considerations.  Specifically,  PEAM  will  be 
compared  with  the  existing  paper-based  documentation  for  the  NATO  SEASPARROW 
Surface  Missile  System  (NSSMS). 


PART  I:  PERFORMANCE  EVALUATION 

Proceduralized  information  provided  by  the  maintenance  requirement  cards  (MRCs)  is 
usually  not  available  to  support  complex  problems  affecting  Navy  hardware  such  as 
multiple  faults  or  cascading  or  intermittent  failures.  When  attempting  to  troubleshoot 
and  correct  system  failures,  Navy  technicians  often  must  consult  and  cross-reference 
maintenance  information  contained  in  the  various  volumes  of  the  ordnance  publications 
(OPs).  These  factors  make  hard-copy  technical  documentation  use  difficult,  confusing, 
and  error-prone. 

Objective 

The  objective  of  Part  1  is  to  compare  the  effectiveness  and  efficiency  of  presenting 
maintenance  information  by  means  of  PEAM  with  that  of  existing  paper-based  documen¬ 
tation  in  supporting  (1)  the  performance  of  procedural  maintenance  on  the  NSSMS  in  a 
fleet  environment  and  (2)  the  detection  and  isolation  of  faults  which  adversely  affect 
normal  NSSMS  operations. 

Task  Description 

Two  types  of  maintenance  tasks  will  be  used  in  the  evaluation:  (1)  procedural  tasks, 
which  consist  of  a  series  of  regularly  scheduled  equipment  checks  and  adjustments  to 
ensure  the  operational  readiness  of  the  equipment  and  to  prevent,  insofar  as  possible, 
unexpected  malfunction  or  failure  of  the  equipment,  and  (2)  troubleshooting  tasks,  which 
are  generally  initiated  as  a  result  of  a  malfunction  indication  on  an  equipment  console  or 
as  a  result  of  a  failed  procedural  maintenance  checkout  step  not  correctable  by  an 
immediately  identifiable  adjustment. 

Participants 

Two  groups  of  personnel  will  be  used  as  test  participants.  One  group  will  consist  of 
16  NSSMS  "C"  school  students  who  have  completed  that  part  of  the  curriculum  that  covers 
the  experimental  tasks.  This  group  will  be  considered  trained,  but  inexperienced.  The 
second  group  will  consist  of  16  NSSMS  "C"  school  graduates  who  have  been  in  the  fleet 
working  on  NSSMS  for  at  least  one  year.  This  group  will  be  considered  trained  and 
experienced. 


■j 


* 


A-l 


Experimental  Design 

The  independent  variables  of  interest  are:  technical  information  delivery  system 
{PEAM  and  paper),  technician  experience  (inexperienced  and  experienced),  and  type  of 
maintenance  (procedural  and  troubleshooting). 

Participants  will  serve  as  their  own  controls.  Each  technician  will  perform  two 
comparable  procedural  maintenance  tasks  and  two  troubleshooting  tasks  using  PEAM  for 
one  of  each  and  conventional  paper-based  documentation  for  the  others.  To  avoid  possible 
experimental  bias  due  to  order  effects,  the  sequences  of  task  combinations  will  be 
counterbalanced. 

The  overall  experimental  design  that  will  be  used  to  collect  the  data  will  have  the 
following  structure: 


Table  A-l 

Structure  of  Experimental  Design 


Electronic 
Documentation 
(n  -  64) 

Paper-based 
Documentation 
(n  =  64) 

Troubleshooting 
n  =  64 

Experienced 
technician 
(n  =  16) 

Inexperienced 
technician 
(n  =  16) 

Experienced 
technician 
(n  =  16) 

Inexperienced 
technician 
(n  =  16) 

Procedural 
n  =  64 

Experienced 
technician 
(n*  16) 

Inexperienced 
technician 
(n=  16) 

Experienced 
technician 
(n  =  16) 

Inexperienced 
technician 
(n  =  16) 

Notes. 

1.  n  =  number  of  data  points  contributing  to  each  cell  mean. 

2.  Since  each  technician  will  provide  data  for  four  cells  in  the  design  matrix,  the 
data  could  well  be  correlated:  that  fact  will  be  taken  into  account  in  the  ANOVA 
calculations. 

3.  There  are  several  sources  of  contamination  both  within  and  between  the 
troubleshooting  and  procedural  maintenance  task  comparisons.  For  example,  for  pro¬ 
cedural  maintenance,  the  transit  between  work  center  and  job  site  may  be  materially 
different  aboard  ship  and  at  the  NSSMS  school  or  even  aboard  different  NSSMS  ships. 
Also,  the  number  and  kinds  of  required  actions  may  differ  (and  thus  the  time  required  and 
potential  for  error  may  differ)  between  procedural  maintenance  and  troubleshooting  tasks. 
These,  too,  will  be  accounted  for  before  making  the  comparative  analyses. 


Table  A-2 


Time  Measures:  Procedural  Maintenance 


Time 

Interval  Task  Sequence 


Initiation 

Start  Task  assigned  to  technician 

Preparation 

1.  Obtain  MRC  or  PEAM  unit  and  information  module;  activate  PEAM 

2.  Verify  MRC/information  module  against  Maintenance  Index  Page  (M1P) 

3.  Review  safety  precautions 

4.  Assemble/prepare  tools,  equipment,  materials  for  transport  to  job  site 

5.  Transit  to  job  site 

6.  Set  up  equipment,  etc.,  at  job  site 

Conduct  Procedural  Maintenance  Task 

7.  Perform  sequential  task  steps  as  presented  on  MRC/PEAM;  encounter 
maladjustment;  identify  adjustment  task 

Prepare  for  Adjustment  Task 

8.  Return  to  work  center;  obtain  MRC/information  module  and  verify  cur¬ 
rency  against  MIP;  assemble  tools,  equipment,  materials;  return  to  job  site 
(NOTE:  If  primary  PEAM  module  contains  adjustment  task  information  and 
no  additional  equipment,  etc.,  is  required,  technician  using  PEAM  does  not 
need  to  return  to  work  center.) 

Adjustment  Task 

9.  Perform  adjustment  task 

Continue  Procedural  Maintenance  Task 

10.  Complete  remaining  sequential  procedural  maintenance  task  steps 
Termination 

11.  Pack  up  equipment,  tools,  materials,  PEAM 

12.  Transit  to  work  center 

13.  Stow  MRC/PEAM  and  module,  gear 

14.  Complete  maintenance  log/report  as  required 
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Table  A-3 


Time  Measures:  Troubleshooting 

Time 

Interval  Task  Sequence 

Initiation 

Start  Activate  portable  computer  (PC);  read  problem  symptoms;  obtain  TM/ac- 

tivate  PEAM 

Conduct  Troubleshooting  Task 

1  Search  TM/PEAM  for  desired  information;  obtain  information;  decide  test 

to  make;  perform  test  (designate  to  PC);  read  result  on  PC 

2a,  b...n  Repeat  I  until  fault  tentatively  located 
3  Identify  part  number  and  "replace"  (enter  number  into  PC) 

k 


Termination 

Report  maintenance  action  taken;  initiate  part  restocking  (enter  into  PC) 


Table  A-4 


Frequency  Counts 


Procedural  Maintenance  Tasks 

Troubleshooting 

(a) 

Steps  performed 

(a) 

Steps  performed 

(b) 

Steps  performed  incorrectly  ^ 

(b) 

Steps  performed  incorrectly^ 

(c) 

Required  steps  not  performed 

(c) 

Inappropriate  test  performed 

(d) 

Non-required  steps  performed 

<d) 

False  removals 

(e) 

2 

Steps  performed  out  of  sequence 

(e) 

Number  of  references  to  documentation 

(f) 

Number  of  references  to  docu- 

(f) 

Number  of  reference  errors5 

mentation  (The  number  of  steps 
required  by  the  documentation 

(g) 

Number  of  uses  of  voice  input  (PEAM) 

will  also  be  noted.) 

(h) 

Number  of  uses  of  speech  output  (PEAM) 

<g> 

Number  of  uses  of  voice  input 
(PEAM) 

(h) 

Number  of  uses  of  speech  output 
(PEAM) 

^ther  than  as  specified  in  documentation. 


2 

Other  than  in  order  specified  in  documentation;  if  the  specified  order  is  next 
resumed,  only  one  count  will  be  recorded,  otherwise  additional  counts  will  accumulate 
until  correct  sequence  is  resumed;  if  the  validity  of  a  step  is  contingent  on  a  skipped  step, 
it  will  be  counted  as  both  incorrect  and  out  of  sequence. 

^Incorrect  steps  include  faulty  readings  (e.g.,  volts  versus  ohms;,  readings  at  wrong 
test  point,  acceptance  of  an  out-of-toJerance  reading  o:  rejector,  of  an  in-tolerance 
reading,  etc. 

4 

A  test  is  inappioj.-  ’tc  if  i\  -'■ovidcs  diagnostic  information  at  the  time  it 

is  made. 

^Includes  reference  to  inappropriate  listing,  directory,  chart,  etc.;  incorrect  ident¬ 
ification  of  line  item,  test  point  number/location;  incorrect  test  value,  part  number,  etc. 
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As  presented  above,  technician  performance  will  be  measured  in  several  ways. 
Analyses  will  be  performed  as  necessary  to  test  the  various  hypotheses  on  the  several 

dimensions.'  The  design  of  the  experiment  will  permit  the  use  of  a  ?.  x  2  x  2  analysis  of 
variance  (ANOVA)  for  many  of  these  tests.  Some  tests,  however,  are  not  amenable  to  the 
2x2x2  ANOVA  approach.  For  example,  a  2x2x2  ANOVA  cannot  be  used  for 
comparisons  of  the  number  of  steps  taken.  There  is  no  a  priori  minimum,  maximum,  or 
optimum  number  of  steps  for  the  troubleshooting  tasks.  There  are  a  specified  number  of 
required  steps  for  the  procedural  maintenance  tasks,  I  jwever.  The  number  of  steps 
actually  taken  in  the  troubleshooting  tasks  may  be  the  sa  ne,  fewer,  or  more  than  those  in 
the  procedural  maintenance  tasks.  There  is  no  common  j^is  for  comparing  them,  they 
come  essentially  from  two  separate  statistical  universes,  and  the  2x2x2  ANOVA  is 
inappropriate.  The  design  of  the  experiment  does,  however,  permit  hypothesis  testing 
using  a  2  x  2  ANOVA,  and  this  will  be  performed.  Where  the  ANOVA  designs  are 
inappropriate,  for  example,  the  frequency  of  use  for  the  PEAM  voice  input,  correlational 
analyses  will  be  used. 


User  Questionnaires 

The  performance  quantitative  measures  provide  adequate  data  to  evaluate  the 
hypotheses  concerning  the  effects  of  the  Information  delivery  systems  on  technicians' 
performance.  However,  these  measures  do  not  address  the  strengths  and  weaknesses  of 
design  features  of  the  delivery  systems  nor  the  acceptance  issues  associated  with  the 
introduction  of  a  new  device.  To  gain  this  complementary  information,  a  user  question¬ 
naire  will  be  administered  to  the  participants  following  their  use  of  both  the  paper-based 
system  and  PEAM.  The  answers  obtained  will  help  evaluate  the  physical  layout  of  the 
displays  and  controls,  the  quality  of  the  visual  and  aural  displays,  the  format  and 
completeness  of  the  information,  and  the  potential  acceptance  of  the  delivery  devices. 
Information  provided  by  this  questionnaire  along  with  the  quantitative  performance  data 
will  also  provide  a  sound  basis  upon  which  to  compare  PEAM  with  MRCs  and  OPs. 

PART  0:  ASSESSMENT  OP  FLEET  IMPLEMENTATION  ISSUES 

Typically,  there  are  many  problems  associoted  with  the  introduction  of  new  equip¬ 
ment  into  the  fleet  and  some  of  these  problems  have  the  potential  of  critically  affecting 
the  acceptance  of  an  innovation  by  members  of  the  intended  user  group.  Problems  have 
to  be  identified  and  the  actions  that  should  be  taken  to  alleviate  or  overcome  them 
described. 

The  implementation  issues  addressed  arc:  (1)  user  considerations,  (2)  logistical 
support  considerations,  (3)  training  considerations,  and  (U)  interface  considerations.  While 
It  Is  recognized  that  these  Issues  focus  on  the  N55M5,  it  is  anticipated  that  many  of  the 
findings  obtained  during  this  phase  will  be  generaiizable  to  other  Navy  systems  and 
equipments  that  could  be  supported  by  an  electronically-based  maintenance  Information 
delivery  system  such  as  PEAM. 

Objective 

The  objective  of  Part  11  Is  to  use  structured  interviews  and  observation  forms  and 
pertinent  Navy  and/or  contractor-prepared  documentation  to  obtain  the  Information 
needed  to  address  fleet  implementation  issues. 
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User  Considerations 


The  acceptance  of  PEAM  by  N5SMS  maintenance  technicians  can  be  facilitated  by 
assessing  factors  of  concern  to  these  personnel.  Accordingly,  this  section  discusses  the 
types  of  questions  and  problems  that  need  to  be  addressed  with  respect  to  the  useis. 

1.  How  many  PEAM  devices  will  be  needed  to  support  shipboard  maintenance  of  the 
NSSMS?  Assessments  will  be  conducted  to  determine  the  number  of  PEAM  devices 
needed  to  support  maintenance  of  the  NSSMS  in  an  efficient  manner. 

2.  Where  should  the  PEAM  device(s)  and  associated  software  be  located  aboard 
ship?  Assessments  will  be  conducted  to  determine  the  optimal  storage  location(s)  for 
PEAM  and  its  associated  memory  modules.  Procedures  would  have  to  be  developed  to 
ensure  that  the  PEAM  device  and  memory  modules  were  returned  to  their  proper  location 
when  not  in  service  and  to  ensure  that  required  preventive  maintenance  actions  were 
performed  on  the  devices.  For  these  reasons,  careful  consideration  must  be  given  to  the 
tradeoffs  between  centralized  versus  decentralized  storage  of  the  PEAM  devices  and 
modules  aboard  ship. 

3.  What  is  required  to  prepare  PEAM  for  maintenance  activities?  In  the  context  of 
the  Navy's  planned  maintenance  system,  a  schedule  of  preventive  maintenance  is 
established  and  reviewed  daily  to  determine  what  tasks  have  to  be  performed.  Accord¬ 
ingly,  information  will  be  obtained  regarding  how  such  a  system  should  oe  structured, 
including  such  possible  approaches  asi  (1)  organizing  modules  according  to  major  system 
functions  (e.g.,  both  preventive/corrective  maintenance  information  for  major  compon¬ 
ents  of  the  NSSMS  stored  on  the  same  memory  module),  (2)  placing  labels  on  the  outside 
of  the  modules  that  provide  a  general  overview  of  their  contents,  and/or  (3)  developing  a 
master  Index  of  all  NSSMS  technical  Information  available  from  PEAM  on  a  separate 
memory  module. 

k.  What  problems  are  anticipated  in  transporting  PEAM  to  the  work  site?  It  will  be 
necessary  to  determine  how  easily  PEAM  can  be  transported  from  its  storage  location  to 
various  work  sites.  The  weight  and  bulk  of  the  device  represent  one  of  the  more  serious 
factors  relating  to  potential  resistance  to  change  by  users  (especially  when  compared  with 
existing  sources  of  procedural  information  such  as  the  maintenance  requirement  card). 

5.  What  problems  are  anticipated  in  setting  up  PEAM  for  use  at  the  work  site? 
Clearly,  the  loading  of  the  module  at  the  storage  location,  transporting  the  device  to  the 
work  site,  and  setting  up  PEAM  for  operation  require  more  expenditure  of  effort  than 
uould  be  required  for  the  current  paper-based  technical  information  delivery  system. 
This  state  of  affairs  raises  the  question  as  to  whether  or  not  the  benefits  of  using  PEAM 
outweigh  the  extra  effort  involved  in  its  use. 

6.  What  problems  are  anticipated  in  the  day-to-day  u^e  of  PEAM?  Here,  considera¬ 
tion  must  be  given  to  the  following  factors: 

a.  Potential  safety  problems  that  may  result  from  connecting  cables  between 
the  device,  ship's  power,  and/or  the  DC  battery  packj  as  well  as  connections  between  the 
headset  and  the  device  (e.g.,  on  or  near  rotating  machinery,  or  near  high  voltage  power 
supplies). 

b.  The  vulnerability  of  PEAM  to  breakage/damage  from  being  dropped. 
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c.  The  practicality  of  the  audio  input/output  features  in  a  high-noise  environ¬ 
ment  (e.g.,  the  flight  deck  during  flight  operations,  in  or  near  machinery  spaces). 

d.  Interference  produced  by  wearing  the  headset  for  the  PEAM  device  at  the 
same  Time  sound-powered  headphones/ear  protective  gear  are  worn  by  the  technician. 

Logistical  Support  Considerations 

The  following  represent  the  types  of  questions  and  problem  areas  that  will  need  to  be 
addressed  regarding  logistical  support  for  the  PEAM  device  and  its  associated  software. 

1.  How  will  updating/correction  of  the  PEAM  software  be  accomplished?  Because 
the  PEAM  software  will  consist  of  exchangeable,  plug-in  memory  modules,  it  is  necessary 
to  determine  how  often,  and  when,  corrections,  revisions,  and  updates  will  be  needed. 
Considering  that  PEAM  represents  an  entirely  new  concept  for  delivering  technical 
information,  methods  must  be  developed  for  periodically  updating  and  revising  the  PEAM 
software  to  reflect  the  addition  of  new  equipments  or  maintenance  procedures,  modifica¬ 
tions  to  existing  equipment  or  procedures,  or  to  delete  these  portions  of  the  data  base 
that  are  obsolete  or  have  been  superseded. 

2.  What  are  the  storage/safeguard  requirements  for  the  PEAM  device  and  associat¬ 
ed  modules?  An  assessment  of  the  physical  storage  requirements  for  the  PEAM  device 
and  memory  modules  will  be  conducted.  This  assessment  will  entail  comparing  the 
storage  requirements  for  the  estimated  number  of  PEAM  devices  and  memory  modules 
with  those  for  current  NSSMS  maintenance  documentation. 

3.  What  sort  of  shipboard  backup  capability  is  needed  to  support  PEAM?  Because  it 
is  very  likely  that  the  PEAM  memory  modules  could  be  damaged  or  lost  in  day-to-day 
usage,  consideration  must  be  given  to  providing  some  sort  of  shipboard  backup  system. 

<i.  Who  will  be  responsible  for  the  maintenance  of  and  spare  parts  supply  for  PEAM? 
An  issue  of  critical  importance  will  be  to  determine  who  will  have  responsibility  for 
repairing  PEAM  at  the  organizational  (i.e.,  ship's  force)  level  and,  similarly,  who  will  have 
responsibility  for  ensuring  that  repair  parts  and  replacement  components  are  readily 
available. 

3.  Should  a  ”prlnt-on-demand"  capability  be  provided  for  PEAM?  Assessments  will 
be  conducted  to  determine  both  the  need  for  as  well  as  the  feasibility  and  practicality  of 
providing  a  "print-on-demand"  capability  for  information  contained  in  the  PEAM  data 
base. 

Training  Considerations 

This  section  discusses  questions  and  problem  areas  regarding  operator  training  for 
PEAM  as  well  as  other  types  of  training  that  could  be  supported  by  it. 

1.  How  will  NSSMS  technicians  learn  to  operate  PEAM?  PEAM  appears  to  be  an 
ideal  candidate  for  the  inclusion  of  an  embedded  training  capability  wherein  technicians 
could  learn  the  basic  operation  and  use  of  the  device  on  line. 

2.  What  other  types  of  training  could  be  supported  by  PEAM?  Another  aspect  of 
this  embedded  training  capability  deals  with  the  potential  PEAM  rnay  have  to  complement 
shipboard  training  prograr  s  that  are  currently  being  conducted  in  the  areas  of  preventive 
and  corrective  maintenance. 
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Interface  Considerations 

The  following  represent  the  types  of  questions  and  problem  areas  that  will  need  to  be 
addressed  with  respect  to  interfacing  PEAM  with  existing  or  developing  maintenance/lo¬ 
gistical  support  systems. 

1.  Can  PEAM  be  interfaced  with  state-of-the-art  diagnostic  capabilities?  The 
feasibility  of  interfacing  PEAM  with  "smart"  built-in  test  (BIT)  equipment  or  devices  that 
provide  an  automated  fault  detection/isolation  capability  will  be  assessed  in  the  course  of 
this  evaluation.  Conceivably,  PEAM  could  be  designed  to  be  patched  into  the  above  types 
of  equipments  and  devices  to  provide  it  with  an  on-line  diagnostic  capability,  or 
alternatively,  the  diagnostic  algorithms  for  these  equipments  and  devices  could  be 
downloaded  onto  PEAM  memory  modules. 

2.  Can  PEAM  be  interfaced  with  state-of-the-art  word  processing  capabilities? 
Assessments  will  be  conducted  to  determine  both  the  need  for,  and  feasibility  of, 
interfacing  a  unit  with  PEAM  that  would  automate  the  preparation  and  processing  of 
standard  Navy  maintenance  reports  and  forms  (e.g.,  OPNAV  4790. 2-series  reports,  supply 
requisition  forms,  and  reports  documenting  accomplishment  of  scheduled  maintenance 
activities). 
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EXAMPLES  OF  TROUBLESHOOTING  INFORMATION  FRAMES 
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This  appendix  represents  examples  of  selected  troubleshooting  information  frames  used  in  conjunction  with  the  director 

control  unit  printed  circuit  card  failure. 1  The  specific  malfunction  involved  is  the  "open"  pin  (XA 18-4)  on  the  7/8A  IA 18  bite 
off-line  oscillator  circuit. 
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The  frames  in  this  appendix  represent  approximately  2  percent  of  the  984  troubleshooting  frames  that  comprise  the 
N55MS  data  base;  thus,  they  are  intended  to  be  representative,  but  not  exhaustive.  In  addition,  each  example  frame  has 
been  photo-reduced  by  two-thirds  of  its  original  5x7  inch  (12.7  x  17.8  cm)  display  size  on  the  Sun  model  3/75  computer. 
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USER  EVALUATION  QUESTIONNAIRE 


Introduction 

Because  PEAM  is  still  undergoing  development,  it  is  important  to  obtain  an  evaluation 
of  the  current  model  from  Navy  technicians  who  have  had  some  'hands  on*  experience  with  it. 
This  questionnaire  Is  designed  for  that  purpose.  You  are  asked  to  evaluate  the  strengths  and 
weaknesses  of  PEAM  in  terms  of  its  (1)  physical  features,  (2)  information  presentation, 
and  (3)  overall  contribution  to  task  performance  as  compared  to  current  maintenance  docu¬ 
ments  to'  the  NSSMS. 

Specifically,  we  ask  you  to  evaluate  the  questionnaire  Hems  using  the  6-polnt  scale 
appearing  to  the  right  of  the  Hems.  Rate  each  hem  by  placing  an  'X"  in  the  appropriate 
column,  we  enoourage  you  to  respond  to  as  many  of  the  questionnaire  Hems  as  possible  but 
recognize  that  there  may  6e  some  items  you  cannot  evaluate  based  on  your  limited  experience 
with  the  PEAM  device.  In  those  cases,  place  an  ’X'  In  the  column  headed:  "Can't  Evaluate*. 


Section  1:  Physical  Features 

Items  1  through  10  concern  the  display  screen. 


Scale  Values 

Items 

Unsatis¬ 

factory 

Marginal 

Satisfactory 

Highly 

Satisfactory 

Outstanding 

Can! 

Evaluate 

1 .  Ease  of  operating  mouse  control 

2.  indication(s)  that  mouse  command  had  been 
activated 

■ 

3.  Adequacy  of  frame  size  for  displaying  information 

4.  Spacing  of  information  on  the  display  screen 
(l.e.,  lack  of  clutter/crowding) 

5.  Brightness  of  the  display 

6.  Contrast  between  information  on  the  display 
and  background 

7.  Legibility  of  displayed  letters  and  words 

8.  Forinat/arrangement  of  graphic  displays  (e.g., 
schematics,  block  diagrams,  IPBs) 

9.  Resolution  and  clarity  of  graphic  displays 

1 0.  Adequacy  of  detail  on  graphic  displays 

Section  2: 


Operation  and  Software  Features 

items  11  through  17  ooncern  the  following  operation  and  software  features: 

•  Organization  and  arrangement  of  maintenance  information 
e  Ease  of  obtaining  relevant  maintenance  information 
t  Extent  to  which  computer  Is  "user-friendly" 


Scale  Values 

Items 

Unsatis¬ 

factory 

Marginal 

Satisfactory 

Highly 

Satisfactory 

Outstanding 

Canl 

Evaluate 

1 1 .  Adequacy  of  the  organization  and  arrangement 
of  maintenance  Information 

12.  Adequacy  of  options  available  on  menus 

13.  Ease  of  using  menus  to  obtain  different  types  of 
maintenance  Information  (e.g.,  MODE/FID 
diagrams,  pans  information) 

14.  Ease  of  obtaining  specific  information  within  a 
particular  type  of  maintenance  information 
(e.g.,  locating  a  specific  part  In  the  parts  info.) 

15.  Adequacy  of  troubleshooting 
diagrams 

16.  Adequacy  of  Information  for  supporting 
maintenance  tasks  (i.e.,  completeness, 
accuracy,  relevancy) 

1 7.  Adequacy  of  "prompts"  on  the  displays  for 
assistlng/guid'  g  the  operator 
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Section  3:  Comparative  Aasessment  (Computer  versus 

Paper  Documentation) 

This  section  of  the  questionnaire  deals  with  the  efficiency  and  effectiveness  of  the  computer 
as  compared  to  current  maintenance  documents  for  the  NSSMS,  such  as  MRCs,  UMRCs,  and  OPs. 
Note  that  the  words  listed  under  the  heading  scale  values  have  changed.  Please  review  this 
scale  carefully  before  rating  the  items. 

To  avoid  repetition  in  the  wording  of  the  items  oor.tained  in  this  section,  begin  each  with  the 
phrase: 


Compared  to  current  maintenance  documents  for  the  NSSMS,  how  would 
you  rate  the  computer  in  terms  of  ... 


Scale  Values 

Items 

Significantly 

Less 

Slightly 

Less 

No 

Difference 

Slightly 

More 

Significantly 

More 

Cant 

Evaluate 

18.  The  overall  time  and  effort  required  to  obtain 
maintenance  information 

19.  The  fatigue  you  experience  when  using  it 

20.  The  oonfusion  or  frustration  you  experience  in 
obtaining  needed  maintenance  information 

21.  The  extent  to  which  it  represents  improvement 
in  the  overall  organization  and  arrangement  of 
maintenance  information 

22.  The  extent  to  which  It  represents  improvement 
in  obtaining  needed  maintenance  information 

23.  The  extent  to  which  it  represents  improvement 

In  the  presentation  of  maintenance  information 

24.  The  extent  to  which  it  represents  Improvement 
in  the  overall  completeness,  accuracy,  and 
applicability  of  maintenance  information 

25.  The  extent  to  which  it  represents  improvement 
in  supporting  maintenance  on  the  NSSMS 
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Section  4: 


General  Reactions  and  Comments 


The  spaces  below  are  provided  for  making  any  comments,  complaints,  suggestions,  etc.,  you  have 
regarding  the  current  model  of  the  PEAM  device. 


This  concludes  the  user  evaluation  questionnaire.  Your  assistance  in  providing  this  essential 
information  is  appreciated. 
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APPENDIX  D 


STRUCTURED  INTERVIEW  QUESTIONS  AND  RESPONSES 


Section  1 

Interview  Questions  Regarding  User  Considerations* 

The  first  two  questions  deal  with  the  number  of  PEAM  devices  needed  to 
support  maintenance  of  the  SEASPARROW  In  an  efficient  manner. 


1.  If  PEAM  were  the  only  source  of  technical  information,  how  many  PEAM  devices  do 
you  think  would  be  needed  to  maintain  a  single  SEASPARROW  system? 


Number  of  Devices 

Totals 

One 

Two 

Three 

Four 

Five 

cv 

0 

6 

2 

2 

1 

11 

DD  963 

1 

9 

6 

1 

2 

19 

Totals 

1 

10 

8 

3 

3 

25 

!?.  Can  you  give  some  of  the  reasons  why  you  think  this  number  would  be  adequate  to  | 

support  the  maintenance  requirements  for  a  single  SEASPARROW  system?  i 


CV 

DD  963 

Totals 

a. 

Convenience  of  multiple  locations 

3 

9 

12 

b. 

Backup  capability 

3 

9 

7 

c. 

Can  have  multiple  jobs  going  on  at  same  time 

5 

0 

5 

Totals 

11 

13 

29 

: 

*The  totals  reported  for  individual  questions  do  not  always  sum  to  25  (i.e.,  the 
number  of  technicians  included  in  the  sample).  In  some  cases,  technicians  simply  did  not 
respond  to  questions,  while  in  others,  they  provided  multiple  responses-- each  of  which 
was  counted  separately. 
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The  next  two  questions  concern  the  physical  placement  of  PEAM  on  board  ship. 


3.  If  it  is  determined  that  one  PEAM  device  is  needed  per  SEASPARROW  system,  where 
do  you  think  it  should  be  located  and  why? 


CV 

DD  963 

Totals 

Why? 

a. 

Computer  room 

0 

13 

13 

Central  location. 

b. 

Control  room/work  center 

s 

0 

8 

Central  location. 

c. 

Transmitter  room 

2 

0 

2 

Most  vital  equipment; 
frequent  problems. 

d. 

Radar  control  room 

1 

0 

1 

Totals 

11 

13 

24 

4.  If  it  is  determined  that  two  or  more  PEAM  devices  are  needed  per  SEASPARROW 
system,  the  devices  could  be  located  in  a  central  location  or  at  different  locations  on 
board  ship.  Which  type  of  arrangement  do  you  think  would  be  best  and  why? 


CV 

DD  963 

Totals 

Why? 

Centralized 

3 

3 

8 

Accountability,  convenience,  keep 
together  with  tools,  other  aids. 

Decentralized 

8 

9 

17 

More  convenient,  better  survival  in  case 
of  attack. 

Totals 

11 

14 

23 

The  next  five  questions  are  based  on  the  assumption  that  the  overall  PEAM  data 
base  for  the  SEASPARROW  system  will  consist  of  several  memory  modules — 
perhaps  as  many  as  10  or  more.  To  a  certain  extent,  this  requirement  would  be 
similar  to  the  library  of  program  tapes  needed  to  support  SEAT  equipment. 
Accordingly,  we  are  interested  in  determining  where  these  modules  might  be 
located  and  how  they  might  be  organized  so  that  SEASPARROW  technicians 
could  locate  needed  maintenance  information  quickly  and  efficiently. 

5.  If  only  one  set  of  PEAM  modules  were  provided  for  the  SEASPARROW  system,  where 
do  you  think  it  should  be  located  and  why? 


CV 

DD  963 

Totals 

Same  location  as  PEAM  device 

3 

1 

4 

Control  room/computer  room 

8 

13 

21 

Totals 

11 

14 

23 

6.  If  two  or  more  sets  of  memory  modules  were  provided  for  PEAM,  they  could  be 
placed  in  one  central  location  or  placed  at  different  locations  onboard  ship.  Which  type  of 
arrangement  do  you  think  would  be  better  and  why? 


CV 

DD  963 

Totals 

Why? 

Centralized 

2 

7 

9 

Located  with  devices. 

Decentralized 

8 

6 

14 

Convenience,  one  set  with  each  device; 
better  survival  chances  in  case  of  attack. 

Totals 

10 

13 

23 

7.  Several  approaches  could  be  used  for  organizing  and  storing  SEASPARROW  mainte¬ 
nance  information  on  PEAM  memory  modules.  For  example,  a  "functional''  approach 
could  be  used  in  which  ali  relevant  information  for  each  subsystem  of  the  SEASPARROW 
(e.g.,  the  director)  would  be  stored  on  one  memory  module.  Alternatively,  a  "systems" 
approach  could  be  used  in  which  FOIDs  for  the  entire  SEASPARROW  system  could  be 
placed  on  one  module,  1PB  information  on  another,  and  so  on.  Which  approach  do  you 
think  would  be  better  and  why? 


CV 

DD  963 

Totals 

Why? 

Functional 

10 

7 

17 

Convenience;  don’t  have  to  search  for 

approach 

each  unit;  used  to  having  arranged  this 
way. 

Systems  approach 

0 

6 

6 

Both  approaches 

1 

0 

1 

Use  functional  approach  for  FOIDs  but 
other  approach  for  system-level  testing/ 

configuration  management  information. 

Totals 

11 

13 

24 

8.  What  sort  of  labeling  do  you  think  should  be  put  on  the  outside  of  the  PEAM  memory 
modules  to  provide  a  general  overview  of  their  contents? 


CV 

DD  963 

Totals 

Identify  by  function/subsystem 

9 

11 

20 

Identify  by  system 

0 

2 

2 

Identify  by  reference  number— similar  to  tech,  manuals 

2 

0 

2 

Totals 

11 

13 

24 
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9.  If  the  maintenance  information  for  the  entire  SEASPARROW  system  requires  JO  or 
more  modules,  what  type  of  indexing  or  cross-referencing  system  do  you  think  would  be 
best  for  locating  specific  information  contained  on  each  module? 


CV 

DD  963 

Totals 

Table  of  contents  for  all  memory  modules  stored  on  a 
separate  module 

5 

9 

14 

Each  module  applied  to  specific  unit— then  broken  down 
by  chapter  and  subsection 

4 

3 

7 

Totals 

9 

12 

21 
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Section  2 


Interview  Questions  Regarding 
Logistical  Support  Considerations 

The  next  three  questions  deal  with  facilities  for  updating,  correcting,  and 
reproducing  technical  information  stored  on  PE  AM  memory  modules. 

10.  One  method  for  updating  and  correcting  the  PEAM  modules  would  be  to  assign  this 
responsibility  to  shore-based  activities  only.  Another  would  be  to  provide  the  necessary 
facilities  to  perform  authorized  reprogramming  of  the  modules  onboard  ship.  Which 
method  do  you  think  would  be  better  and  why? 


CV 

DD  963 

Totals 

Why? 

Shipboard 

5 

7 

12 

Shipboard  techs— know  what  needs 
updating;  can  be  accomplished  quicker 
and  more  efficiently;  accountability. 

Shore-based 

command 

6 

6 

12 

Shore  activity  can  research  first  and 
send  to  ail  ships  at  once;  standardiza¬ 
tion,  better  quality  control;  fleet  person¬ 
nel  more  apt  to  damage  modules. 

Totals 

11 

13 

24 

1 1.  Because  PEAM  memory  modules  might  be  damaged  or  lost  in  everyday  use,  some  sort 
of  backup  capability  must  be  provided.  One  method  for  obtaining  backup  copies  of  the 
memory  modules  would  be  to  order  them  from  shore-based  activities.  Another  would  be 
to  provide  the  necessary  facilities  to  reproduce  PEAM  memory  modules  onboard  ship. 
Which  method  do  you  think  would  be  better  and  why? 

CV 

DD  963 

Totals 

Why? 

Shipboard 

7 

9 

16 

Replacement  could  be  obtained  quicker; 
save  time,  money,  paperwork. 

Shore-based 

4 

4 

S 

Trust  shore-based  facility  more;  more 
manpower  available. 

Totals 

11 

13 

24 

1 2.  If  the  foregoing  updating,  correcting,  and  backup  capabilities  were  provided  onboard 
ship,  who  do  you  think  should  have  the  responsibility  for  performing  these  duties? 


CV  DD  963  Totals 


5EASPARROW  technicians 
Technical  Library 

Totals 


9  12  21 
1  0  1 


10  12  22 


The  next  two  questions  concern  possible  environmental  conditions  found  on¬ 
board  ship  that  might  damage  or  degrade  the  PEAM  device  and  memory 
modules. 

13.  Do  you  think  that  PEAM  might  be  damaged  or  degraded  by  emissions  from  shipboard 
transmitters?  If  so,  specify  the  location(s)  and  operating  conditions  in  which  you  think 


such  a  problem  might  occur. 

CV 

DD  963 

Totals 

Why? 

Yes 

6 

5 

11 

If  used  near  radars/transmitter;  when 

used  outside  on  launchers  and  directors. 

No 

5 

6 

11 

Totals 

11 

11 

22 

14.  Can  you  think  of  any  other  type(s)  of  environmental  conditions  onboard  ship  that 
might  damage  or  degrade  the  PEAM  device  or  memory  modules?  If  so,  specify  which 
condition(s)  and  what  adverse  effect(s)  you  can  foresee. 


CV 

DD  963 

Totals 

Why? 

Yes 

5 

8 

13 

Rain  and/or  salt  water;  vibration;  high 
temperatures;  loss  of  A/C  current  or 
surge;  dirt. 

No 

6 

4 

10 

Totals 

11 

12 

23 

The  next  two  questions  concern  the  need  for  providing  logistics  support  for  the 
PEAM  device  itself  after  contractor-provided  services  have  been  terminated. 


15.  Responsibility  for  the  maintenance  and  spare  parts  supply  for  PEAM  could  be 
assigned  on  a  departmental  basis;  or  alternatively  assigned  to  a  specific  Navy  rating  or 


work  center.  Which  method  do  you  think  would  be  better? 

CV 

DD  963 

Totals 

Own  department 

8 

9 

17 

Specific  Navy  rating 

2 

2 

4 

Totals 

10 

11 

21 

16.  What  method  do  you  think  would  be  best  for  presenting  the  technical  information 
needed  to  perform  preventive  and  corrective  maintenance  on  the  PEAM  device  itself? 


CV 

DD  963 

Totals 

PMS  cards/technical  manuals 

6 

7 

13 

PEAM  module 

3 

4 

7 

Totals 

9 

11 

20 

The  next  three  questions  concern  the  inclusion  of  a  "print-on -demand" 
capability  for  PEAM. 


17.  Can  you  think  of  any  situations  and/or  conditions  that  would  require  you  to  obtain 
paper  copies  of  the  information  contained  in  the  PEAM  memory  modules?  If  so,  please 
specify  what  these  might  be. 


CV 

DD  963 

Totals 

Why? 

Yes 

5 

12 

17 

Have  information  to  CASREP  a  system; 
use  for  bookmark  in  troubleshooting 
process;  permit  2  techs,  to  work  at 
same  time;  maintain  fault  histories— 
record  keeping;  use  for  training;  taking 
into  remote  locations. 

No 

5 

1 

6 

Totals 

10 

13 

23 

18.  If  a  print-on-demand  capability  were  provided,  where  do  you  think  the  printer  unit 
should  be  located? 


CV 

DD  963 

Totals 

Where  PEAM  is  stored 

5 

0 

3 

Mounted  somewhere  in  control  space/computer  room 

4 

9 

13 

Don't  need  capability 

2 

0 

2 

Totals 

11 

9 

20 
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19.  Although  there  may  be  advantages  to  having  a  print-on-demand  capability,  can 
you  foresee  my  problems  with  its  use?  If  so,  please  specify. 


CV 

DD  963 

Totals 

Have  to  do  PMS  <Sc  repair/maintain  another  piece  of 
equipment 

7 

3 

10 

Infrequent  use,  costly 

? 

0 

2 

No  paper/ink  supplies  in  stock 

I 

2 

3 

No  problems 

2 

9 

11 

Totals 

12 

14 

26 

Section  3 


Interview  Questions  Regarding  Training  Considerations 

The  questions  in  this  section  deal  with  methods  for  teaching  NSSMS  technicians 
how  to  operate  PEAM  and  its  potential  for  supporting  other  types  of  shipboard 
training. 

20.  What  do  you  think  would  be  the  best  method  for  presenting  the  information  needed  to 
learn  the  basic  operation  of  PEAM  on  board  ship? 


CV 

DD  963 

Totals 

Demonstration/instruction  book 

3 

5 

10 

PEAM  module 

1 

0 

1 

Have  work  center  supervisor  teach  others  by  COT 

1 

4 

5 

Using  PMS  cards 

1 

0 

1 

School  course 

0 

4 

4 

Totals 

s 

13 

21 

21.  Do  you  think  that  preliminary  training  on  the  operation  of  PEAM  should  be  provided 
in  Class  "A"  or  "C"  schools?  If  so,  specify  which  school  you  think  would  be  the  most 
appropriate. 


CV 

DD  963 

Totals 

"A"  School 

1 

0 

1 

"C"  School 

6 

11 

17 

Both 

3 

0 

3 

Not  in  school 

1 

1 

2 

Totals 

11 

12 

23 

22.  Are  there  any  other  shipboard  training  applications  that  you  think  PEAM  could 
support?  If  so,  please  specify  what  type  of  training  this  might  include. 


CV 

DD  963 

Totals 

General  damage  conti  <l/general  maintenance  training 

1 

2 

3 

EPICS  training 

1 

2 

3 

All  fire  control  equipment 

1 

2 

3 

Any  electronic  troubleshooting 

1 

0 

1 

Deck  preservation 

0 

I 

1 

None 

5 

1 

6 

Totals 

9 

S 

17 

D-9 


Section  4 


Interview  Questio.  s  Regarding  Interface  Considerations 

The  last  two  questions  deal  with  the  need  for  interfacing  PEAM  with  other 
devices  to  expand  its  present  capabilities  and  functions, 

23.  Are  there  any  functions,  capabilities,  or  procedures  for  diagnosing  faults  in  the 
SEASPARROW  system  that  you  would  like  to  see  interfaced  with  the  PEAM  device?  If  so, 
please  specify. 


CV 

DD  963 

Totals 

Capability  for  feeding  symptoms  into  PEAM  and  have  it 
diagnose  fault,  and  tell  technician  where  to  begin 

5 

3 

8 

Computer  or  SEAT 

3 

5 

8 

Verbally  giving  requirements  for  proper  operation 

1 

0 

1 

No 

1 

4 

5 

Totals 

10 

12 

22 

24.  Are  there  any  logistical  support  functions  for  the  SEASPARROW  system 

that  are 

currently  performed  manually  that  you  would  like  to  see 

"automated"  by  means  of  a 

PEAM  interface  unit?  If  so,  please  specify  what  these  functions 

might  include. 

CV 

DD  963 

Totals 

Capability  of  punching  in  Ref  Des  and  have  PEAM  tell  me 
part  No.  and  NSN  for  desired  component 

1 

0 

i 

List  of  APLs,  NSNs 

1 

3 

4 

Report  writing 

0 

3 

3 

Voltage  readings 

0 

1 

1 

System  checks  thru  SEAT 

0 

1 

1 

None 

9 

4 

13 

Totals 

11 

IT" 

23 
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